[2023-12-17:11:19:23:INFO:sockeye.utils:log_sockeye_version] Sockeye: 3.1.36, commit unknown, path /home/amoryo/sign-language/signbank-annotation/signbank-plus/signbank_plus/nmt/sockeye/sockeye/sockeye/__init__.py [2023-12-17:11:19:23:INFO:sockeye.utils:log_torch_version] PyTorch: 1.12.1+cu113 (/home/amoryo/.local/lib/python3.10/site-packages/torch/__init__.py) [2023-12-17:11:19:23:INFO:sockeye.utils:log_basic_info] Command: /home/amoryo/sign-language/signbank-annotation/signbank-plus/signbank_plus/nmt/sockeye/sockeye/sockeye/train.py -d /shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/train_data --weight-tying-type none --batch-size 512 --max-seq-len 512:128 --decode-and-evaluate 500 --validation-source /home/amoryo/sign-language/signbank-annotation/signbank-plus/data/parallel/cleaned/dev.source.tokenized --validation-target /shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/dev.target.bpe --optimized-metric chrf --max-num-checkpoint-not-improved 20 --output /shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model --params /shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded/model/params.best [2023-12-17:11:19:23:INFO:sockeye.utils:log_basic_info] Arguments: Namespace(config=None, source=None, source_factors=[], source_factors_use_source_vocab=[], target_factors=[], target_factors_use_target_vocab=[], target=None, end_of_prepending_tag=None, prepared_data='/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/train_data', validation_source='/home/amoryo/sign-language/signbank-annotation/signbank-plus/data/parallel/cleaned/dev.source.tokenized', validation_source_factors=[], validation_target='/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/dev.target.bpe', validation_target_factors=[], no_bucketing=False, bucket_width=8, bucket_scaling=False, max_seq_len=(512, 128), source_vocab=None, target_vocab=None, source_factor_vocabs=[], target_factor_vocabs=[], shared_vocab=False, num_words=(0, 0), word_min_count=(1, 1), pad_vocab_to_multiple_of=8, output='/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model', overwrite_output=False, params='/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded/model/params.best', allow_missing_params=False, ignore_extra_params=False, encoder='transformer', decoder='transformer', num_layers=(6, 6), transformer_model_size=(512, 512), transformer_attention_heads=(8, 8), transformer_feed_forward_num_hidden=(2048, 2048), transformer_feed_forward_use_glu=False, transformer_activation_type=('relu', 'relu'), transformer_positional_embedding_type='fixed', transformer_block_prepended_cross_attention=False, transformer_preprocess=('n', 'n'), transformer_postprocess=('dr', 'dr'), lhuc=None, num_embed=(None, None), source_factors_num_embed=[], target_factors_num_embed=[], source_factors_combine=[], target_factors_combine=[], source_factors_share_embedding=[], target_factors_share_embedding=[], weight_tying_type='none', dtype='float32', clamp_to_dtype=False, amp=False, apex_amp=False, neural_vocab_selection=None, neural_vocab_selection_block_loss=False, batch_size=512, batch_type='word', batch_sentences_multiple_of=8, update_interval=1, label_smoothing=0.1, label_smoothing_impl='mxnet', length_task=None, length_task_weight=1.0, length_task_layers=1, bow_task_weight=1.0, bow_task_pos_weight=10, target_factors_weight=[1.0], optimized_metric='chrf', checkpoint_interval=4000, min_samples=None, max_samples=None, min_updates=None, max_updates=None, max_seconds=None, max_checkpoints=None, max_num_checkpoint_not_improved=20, checkpoint_improvement_threshold=0.0, min_num_epochs=None, max_num_epochs=None, embed_dropout=(0.0, 0.0), transformer_dropout_attention=(0.1, 0.1), transformer_dropout_act=(0.1, 0.1), transformer_dropout_prepost=(0.1, 0.1), optimizer='adam', optimizer_betas=(0.9, 0.999), optimizer_eps=1e-08, dist=False, initial_learning_rate=0.0002, weight_decay=0.0, momentum=0.0, gradient_clipping_threshold=1.0, gradient_clipping_type='none', learning_rate_scheduler_type='plateau-reduce', learning_rate_reduce_factor=0.9, learning_rate_reduce_num_not_improved=8, learning_rate_warmup=0, no_reload_on_learning_rate_reduce=False, fixed_param_strategy=None, fixed_param_names=[], local_rank=None, deepspeed_fp16=False, deepspeed_bf16=False, decode_and_evaluate=500, stop_training_on_decoder_failure=False, seed=1, keep_last_params=-1, keep_initializations=False, cache_last_best_params=0, cache_strategy='best', cache_metric='perplexity', dry_run=False, device_id=0, use_cpu=False, env=None, tf32=True, quiet=False, quiet_secondary_workers=False, no_logfile=False, loglevel='INFO', loglevel_secondary_workers='INFO') [2023-12-17:11:19:23:INFO:__main__:train] Adjusting maximum length to reserve space for a BOS/EOS marker. New maximum length: (513, 129) [2023-12-17:11:19:24:INFO:sockeye.utils:init_device] CUDA: allow tf32 (float32 but with 10 bits precision) [2023-12-17:11:19:24:INFO:__main__:train] Training Device: cuda:0 [2023-12-17:11:19:24:INFO:sockeye.utils:seed_rngs] Random seed: 1 [2023-12-17:11:19:24:INFO:sockeye.utils:seed_rngs] PyTorch seed: 1 [2023-12-17:11:19:24:INFO:sockeye.data_io:get_prepared_data_iters] =============================== [2023-12-17:11:19:24:INFO:sockeye.data_io:get_prepared_data_iters] Creating training data iterator [2023-12-17:11:19:24:INFO:sockeye.data_io:get_prepared_data_iters] =============================== [2023-12-17:11:19:24:INFO:sockeye.vocab:vocab_from_json] Vocabulary (1232 words) loaded from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/train_data/vocab.src.0.json" [2023-12-17:11:19:24:INFO:sockeye.vocab:vocab_from_json] Vocabulary (5976 words) loaded from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/train_data/vocab.trg.0.json" [2023-12-17:11:19:24:INFO:sockeye.data_io:log] Tokens: source 15626084 target 2020240 [2023-12-17:11:19:24:INFO:sockeye.data_io:log] Number of tokens: source 1 target 116 [2023-12-17:11:19:24:INFO:sockeye.data_io:log] Vocabulary coverage: source 100% target 100% [2023-12-17:11:19:24:INFO:sockeye.data_io:log] 354707 sequences across 65 buckets [2023-12-17:11:19:24:INFO:sockeye.data_io:log] 20 sequences did not fit into buckets and were discarded [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (8, 8): 1 samples in 1 batches of 256, ~512.0 target tokens/batch, trg/src length ratio: 0.33 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (16, 16): 29122 samples in 228 batches of 128, ~523.0 target tokens/batch, trg/src length ratio: 0.28 (+-0.18) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (24, 24): 57389 samples in 513 batches of 112, ~509.1 target tokens/batch, trg/src length ratio: 0.22 (+-0.13) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (32, 32): 123411 samples in 1102 batches of 112, ~516.0 target tokens/batch, trg/src length ratio: 0.17 (+-0.10) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (40, 40): 36766 samples in 354 batches of 104, ~507.4 target tokens/batch, trg/src length ratio: 0.14 (+-0.12) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (48, 48): 44876 samples in 432 batches of 104, ~518.2 target tokens/batch, trg/src length ratio: 0.12 (+-0.12) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (56, 56): 21849 samples in 228 batches of 96, ~506.0 target tokens/batch, trg/src length ratio: 0.10 (+-0.11) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (64, 64): 6758 samples in 77 batches of 88, ~510.7 target tokens/batch, trg/src length ratio: 0.10 (+-0.13) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (72, 72): 8500 samples in 97 batches of 88, ~530.5 target tokens/batch, trg/src length ratio: 0.09 (+-0.16) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (80, 80): 2765 samples in 39 batches of 72, ~492.1 target tokens/batch, trg/src length ratio: 0.10 (+-0.26) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (88, 88): 3715 samples in 59 batches of 64, ~499.4 target tokens/batch, trg/src length ratio: 0.10 (+-0.20) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (96, 96): 2686 samples in 84 batches of 32, ~481.5 target tokens/batch, trg/src length ratio: 0.17 (+-0.26) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (104, 104): 722 samples in 13 batches of 56, ~484.0 target tokens/batch, trg/src length ratio: 0.11 (+-0.32) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (112, 112): 752 samples in 14 batches of 56, ~483.9 target tokens/batch, trg/src length ratio: 0.10 (+-0.31) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (120, 120): 417 samples in 9 batches of 48, ~512.6 target tokens/batch, trg/src length ratio: 0.13 (+-0.39) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (128, 128): 491 samples in 9 batches of 56, ~511.5 target tokens/batch, trg/src length ratio: 0.09 (+-0.31) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (136, 129): 402 samples in 8 batches of 56, ~521.1 target tokens/batch, trg/src length ratio: 0.07 (+-0.06) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (144, 129): 347 samples in 7 batches of 56, ~502.5 target tokens/batch, trg/src length ratio: 0.06 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (152, 129): 249 samples in 6 batches of 48, ~526.7 target tokens/batch, trg/src length ratio: 0.07 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (160, 129): 196 samples in 5 batches of 40, ~538.6 target tokens/batch, trg/src length ratio: 0.09 (+-0.06) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (168, 129): 297 samples in 8 batches of 40, ~529.3 target tokens/batch, trg/src length ratio: 0.08 (+-0.06) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (176, 129): 376 samples in 8 batches of 48, ~543.6 target tokens/batch, trg/src length ratio: 0.07 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (184, 129): 262 samples in 7 batches of 40, ~559.4 target tokens/batch, trg/src length ratio: 0.08 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (192, 129): 246 samples in 8 batches of 32, ~516.6 target tokens/batch, trg/src length ratio: 0.09 (+-0.06) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (200, 129): 213 samples in 7 batches of 32, ~531.5 target tokens/batch, trg/src length ratio: 0.08 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (208, 129): 269 samples in 9 batches of 32, ~525.7 target tokens/batch, trg/src length ratio: 0.08 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (216, 129): 331 samples in 11 batches of 32, ~533.0 target tokens/batch, trg/src length ratio: 0.08 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (224, 129): 317 samples in 14 batches of 24, ~462.2 target tokens/batch, trg/src length ratio: 0.09 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (232, 129): 350 samples in 15 batches of 24, ~518.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (240, 129): 356 samples in 15 batches of 24, ~519.4 target tokens/batch, trg/src length ratio: 0.09 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (248, 129): 368 samples in 16 batches of 24, ~502.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (256, 129): 451 samples in 19 batches of 24, ~551.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (264, 129): 454 samples in 19 batches of 24, ~567.3 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (272, 129): 497 samples in 21 batches of 24, ~580.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (280, 129): 483 samples in 21 batches of 24, ~608.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (288, 129): 519 samples in 33 batches of 16, ~415.3 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (296, 129): 572 samples in 36 batches of 16, ~416.2 target tokens/batch, trg/src length ratio: 0.09 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (304, 129): 500 samples in 32 batches of 16, ~435.5 target tokens/batch, trg/src length ratio: 0.09 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (312, 129): 550 samples in 35 batches of 16, ~431.4 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (320, 129): 608 samples in 38 batches of 16, ~436.8 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (328, 129): 561 samples in 36 batches of 16, ~448.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (336, 129): 537 samples in 34 batches of 16, ~480.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (344, 129): 481 samples in 31 batches of 16, ~479.5 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (352, 129): 568 samples in 36 batches of 16, ~495.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (360, 129): 516 samples in 33 batches of 16, ~487.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (368, 129): 468 samples in 30 batches of 16, ~488.4 target tokens/batch, trg/src length ratio: 0.08 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (376, 129): 554 samples in 35 batches of 16, ~503.9 target tokens/batch, trg/src length ratio: 0.08 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (384, 129): 512 samples in 32 batches of 16, ~520.2 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (392, 129): 486 samples in 31 batches of 16, ~522.2 target tokens/batch, trg/src length ratio: 0.08 (+-0.02) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (400, 129): 455 samples in 29 batches of 16, ~551.4 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (408, 129): 130 samples in 9 batches of 16, ~521.7 target tokens/batch, trg/src length ratio: 0.08 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (416, 129): 2 samples in 1 batches of 96, ~528.0 target tokens/batch, trg/src length ratio: 0.01 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (424, 129): 4 samples in 1 batches of 8, ~344.0 target tokens/batch, trg/src length ratio: 0.10 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:_load_shard] Loading shard /shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/train_data/shard.00000. [2023-12-17:11:19:24:INFO:sockeye.data_io:get_validation_data_iter] ================================= [2023-12-17:11:19:24:INFO:sockeye.data_io:get_validation_data_iter] Creating validation data iterator [2023-12-17:11:19:24:INFO:sockeye.data_io:get_validation_data_iter] ================================= [2023-12-17:11:19:24:INFO:sockeye.data_io:analyze_sequence_lengths] 2845 sequences of maximum length (513, 129) in '/home/amoryo/sign-language/signbank-annotation/signbank-plus/data/parallel/cleaned/dev.source.tokenized' and '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/dev.target.bpe'. [2023-12-17:11:19:24:INFO:sockeye.data_io:analyze_sequence_lengths] Mean training target/source length ratio: 0.16 (+-0.12) [2023-12-17:11:19:24:INFO:sockeye.data_io:log] Tokens: source 128497 target 16716 [2023-12-17:11:19:24:INFO:sockeye.data_io:log] Number of tokens: source 0 target 1 [2023-12-17:11:19:24:INFO:sockeye.data_io:log] Vocabulary coverage: source 100% target 100% [2023-12-17:11:19:24:INFO:sockeye.data_io:log] 2845 sequences across 65 buckets [2023-12-17:11:19:24:INFO:sockeye.data_io:log] 0 sequences did not fit into buckets and were discarded [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (16, 16): 242 samples in 2 batches of 128, ~523.0 target tokens/batch, trg/src length ratio: 0.29 (+-0.15) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (24, 24): 472 samples in 5 batches of 112, ~509.1 target tokens/batch, trg/src length ratio: 0.22 (+-0.11) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (32, 32): 920 samples in 9 batches of 112, ~516.0 target tokens/batch, trg/src length ratio: 0.17 (+-0.10) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (40, 40): 333 samples in 4 batches of 104, ~507.4 target tokens/batch, trg/src length ratio: 0.13 (+-0.07) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (48, 48): 382 samples in 4 batches of 104, ~518.2 target tokens/batch, trg/src length ratio: 0.12 (+-0.09) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (56, 56): 178 samples in 2 batches of 96, ~506.0 target tokens/batch, trg/src length ratio: 0.11 (+-0.15) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (64, 64): 63 samples in 1 batches of 88, ~510.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (72, 72): 50 samples in 1 batches of 88, ~530.5 target tokens/batch, trg/src length ratio: 0.08 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (80, 80): 18 samples in 1 batches of 72, ~492.1 target tokens/batch, trg/src length ratio: 0.11 (+-0.16) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (88, 88): 18 samples in 1 batches of 64, ~499.4 target tokens/batch, trg/src length ratio: 0.14 (+-0.20) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (96, 96): 20 samples in 1 batches of 32, ~481.5 target tokens/batch, trg/src length ratio: 0.21 (+-0.17) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (104, 104): 4 samples in 1 batches of 56, ~484.0 target tokens/batch, trg/src length ratio: 0.08 (+-0.02) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (112, 112): 10 samples in 1 batches of 56, ~483.9 target tokens/batch, trg/src length ratio: 0.07 (+-0.02) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (120, 120): 2 samples in 1 batches of 48, ~512.6 target tokens/batch, trg/src length ratio: 0.05 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (128, 128): 4 samples in 1 batches of 56, ~511.5 target tokens/batch, trg/src length ratio: 0.12 (+-0.08) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (136, 129): 1 samples in 1 batches of 56, ~521.1 target tokens/batch, trg/src length ratio: 0.10 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (144, 129): 2 samples in 1 batches of 56, ~502.5 target tokens/batch, trg/src length ratio: 0.17 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (152, 129): 2 samples in 1 batches of 48, ~526.7 target tokens/batch, trg/src length ratio: 0.15 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (160, 129): 2 samples in 1 batches of 40, ~538.6 target tokens/batch, trg/src length ratio: 0.13 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (176, 129): 5 samples in 1 batches of 48, ~543.6 target tokens/batch, trg/src length ratio: 0.05 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (184, 129): 4 samples in 1 batches of 40, ~559.4 target tokens/batch, trg/src length ratio: 0.08 (+-0.06) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (208, 129): 4 samples in 1 batches of 32, ~525.7 target tokens/batch, trg/src length ratio: 0.04 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (224, 129): 2 samples in 1 batches of 24, ~462.2 target tokens/batch, trg/src length ratio: 0.02 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (232, 129): 8 samples in 1 batches of 24, ~518.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (240, 129): 3 samples in 1 batches of 24, ~519.4 target tokens/batch, trg/src length ratio: 0.09 (+-0.03) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (248, 129): 5 samples in 1 batches of 24, ~502.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (256, 129): 5 samples in 1 batches of 24, ~551.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (264, 129): 8 samples in 1 batches of 24, ~567.3 target tokens/batch, trg/src length ratio: 0.09 (+-0.02) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (272, 129): 4 samples in 1 batches of 24, ~580.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (280, 129): 3 samples in 1 batches of 24, ~608.7 target tokens/batch, trg/src length ratio: 0.07 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (288, 129): 5 samples in 1 batches of 16, ~415.3 target tokens/batch, trg/src length ratio: 0.02 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (296, 129): 4 samples in 1 batches of 16, ~416.2 target tokens/batch, trg/src length ratio: 0.08 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (304, 129): 3 samples in 1 batches of 16, ~435.5 target tokens/batch, trg/src length ratio: 0.12 (+-0.05) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (312, 129): 11 samples in 1 batches of 16, ~431.4 target tokens/batch, trg/src length ratio: 0.09 (+-0.02) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (320, 129): 6 samples in 1 batches of 16, ~436.8 target tokens/batch, trg/src length ratio: 0.08 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (328, 129): 6 samples in 1 batches of 16, ~448.7 target tokens/batch, trg/src length ratio: 0.09 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (344, 129): 8 samples in 1 batches of 16, ~479.5 target tokens/batch, trg/src length ratio: 0.10 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (352, 129): 4 samples in 1 batches of 16, ~495.0 target tokens/batch, trg/src length ratio: 0.09 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (360, 129): 4 samples in 1 batches of 16, ~487.7 target tokens/batch, trg/src length ratio: 0.08 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (368, 129): 2 samples in 1 batches of 16, ~488.4 target tokens/batch, trg/src length ratio: 0.10 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (376, 129): 2 samples in 1 batches of 16, ~503.9 target tokens/batch, trg/src length ratio: 0.08 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (384, 129): 6 samples in 1 batches of 16, ~520.2 target tokens/batch, trg/src length ratio: 0.08 (+-0.04) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (392, 129): 4 samples in 1 batches of 16, ~522.2 target tokens/batch, trg/src length ratio: 0.08 (+-0.00) [2023-12-17:11:19:24:INFO:sockeye.data_io:describe_data_and_buckets] Bucket (400, 129): 6 samples in 1 batches of 16, ~551.4 target tokens/batch, trg/src length ratio: 0.08 (+-0.01) [2023-12-17:11:19:24:INFO:sockeye.data_io:load] Created bucketed parallel data set. Introduced padding: source=7.2% target=85.8%) [2023-12-17:11:19:24:INFO:sockeye.vocab:vocab_to_json] Vocabulary saved to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/vocab.src.0.json" [2023-12-17:11:19:24:INFO:sockeye.vocab:vocab_to_json] Vocabulary saved to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/vocab.trg.0.json" [2023-12-17:11:19:24:INFO:__main__:train] Vocabulary sizes: source=[1232] target=[5976] [2023-12-17:11:19:24:INFO:__main__:get_num_embed] Source embedding size was not set it will automatically be adjusted to match the Transformer source model size (512). [2023-12-17:11:19:24:INFO:__main__:get_num_embed] Target embedding size was not set it will automatically be adjusted to match the Transformer target model size (512). [2023-12-17:11:19:24:INFO:__main__:create_optimizer_config] OptimizerConfig(name='adam', running_on_gpu=True, lr=0.0002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0, momentum=0.0, gradient_clipping_type='none', gradient_clipping_threshold=1.0, update_interval=1) [2023-12-17:11:19:24:INFO:__main__:create_optimizer_config] Gradient accumulation over 1 batch(es) by 1 worker(s). Effective batch size: 512 [2023-12-17:11:19:24:INFO:sockeye.model:__init__] ModelConfig(config_data=DataConfig(data_statistics=DataStatistics(num_sents=354707, num_discarded=20, num_tokens_source=15626084, num_tokens_target=2020240, num_unks_source=1, num_unks_target=116, max_observed_len_source=417, max_observed_len_target=128, size_vocab_source=1232, size_vocab_target=5976, length_ratio_mean=0.16320710693441579, length_ratio_std=0.1371393774100467, buckets=[(8, 8), (16, 16), (24, 24), (32, 32), (40, 40), (48, 48), (56, 56), (64, 64), (72, 72), (80, 80), (88, 88), (96, 96), (104, 104), (112, 112), (120, 120), (128, 128), (136, 129), (144, 129), (152, 129), (160, 129), (168, 129), (176, 129), (184, 129), (192, 129), (200, 129), (208, 129), (216, 129), (224, 129), (232, 129), (240, 129), (248, 129), (256, 129), (264, 129), (272, 129), (280, 129), (288, 129), (296, 129), (304, 129), (312, 129), (320, 129), (328, 129), (336, 129), (344, 129), (352, 129), (360, 129), (368, 129), (376, 129), (384, 129), (392, 129), (400, 129), (408, 129), (416, 129), (424, 129), (432, 129), (440, 129), (448, 129), (456, 129), (464, 129), (472, 129), (480, 129), (488, 129), (496, 129), (504, 129), (512, 129), (513, 129)], num_sents_per_bucket=[1, 29122, 57389, 123411, 36766, 44876, 21849, 6758, 8500, 2765, 3715, 2686, 722, 752, 417, 491, 402, 347, 249, 196, 297, 376, 262, 246, 213, 269, 331, 317, 350, 356, 368, 451, 454, 497, 483, 519, 572, 500, 550, 608, 561, 537, 481, 568, 516, 468, 554, 512, 486, 455, 130, 2, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], average_len_target_per_bucket=[2.0, 4.08629215026444, 4.545226437122044, 4.607174400985307, 4.87920905184137, 4.982440502718691, 5.27049292873817, 5.803196211897, 6.028941176470578, 6.834719710669081, 7.802691790040373, 15.045793000744581, 8.642659279778403, 8.640957446808516, 10.678657074340526, 9.13441955193482, 9.305970149253735, 8.974063400576362, 10.971887550200805, 13.464285714285717, 13.232323232323239, 11.324468085106396, 13.984732824427489, 16.142276422764233, 16.61032863849766, 16.427509293680295, 16.655589123867063, 19.258675078864364, 21.614285714285707, 21.643258426966298, 20.918478260869566, 22.957871396895783, 23.638766519823815, 24.167002012072434, 25.362318840579718, 25.95375722543352, 26.010489510489492, 27.22000000000001, 26.959999999999987, 27.297697368421066, 28.040998217468793, 30.042830540037233, 29.966735966735982, 30.934859154929573, 30.47868217054262, 30.527777777777782, 31.492779783393505, 32.51171874999999, 32.6358024691358, 34.461538461538474, 32.60769230769231, 5.5, 43.0, None, None, None, None, None, None, None, None, None, None, None, None], length_ratio_stats_per_bucket=[(0.3333333333333333, 0.0), (0.28246393697985434, 0.17868752447804973), (0.21840710265332788, 0.1330505772378312), (0.16560142798704922, 0.09581195473826641), (0.13801367492489092, 0.12429965021659338), (0.11863212215522084, 0.1208393385452983), (0.10151133866588294, 0.11072333780515448), (0.09838819717267734, 0.12557601720946082), (0.09394884997066442, 0.16159177653077658), (0.10402554625981722, 0.26083679437294416), (0.1012342945734544, 0.20099512839826167), (0.17281772320739658, 0.257545103018524), (0.10845391475564008, 0.3239959561352876), (0.09935073708696769, 0.3059573403277105), (0.12635011083619693, 0.3907243857496131), (0.08949294838769961, 0.3144956594612652), (0.07007331783529426, 0.05774973922713284), (0.06411727035132861, 0.04726139664600602), (0.07382609782015778, 0.050521761108029695), (0.08590157138956556, 0.05569960282284095), (0.08062786352616935, 0.058790770283216505), (0.06563222282256796, 0.05450274064413921), (0.07765999502184046, 0.05057167990395854), (0.08555873649959676, 0.05960592960682603), (0.08460026909745419, 0.05238594583690578), (0.08041439956489124, 0.04792844419538253), (0.07859928415542815, 0.05072523580179588), (0.08720935617277471, 0.04382289790728185), (0.0945785744419209, 0.05412799726318098), (0.09147639273741005, 0.04623369327444139), (0.08566937441195915, 0.03729904156233976), (0.09089250053653752, 0.03997864088737986), (0.09074467762227163, 0.03321456251239067), (0.09005889312946631, 0.03317108879820214), (0.09171046510420815, 0.035431771966381115), (0.09113688227781093, 0.03302006652634936), (0.08893255045731784, 0.03625826248767216), (0.09050986731593187, 0.03584124694886162), (0.08733945286928807, 0.02963045027026122), (0.08629641342788141, 0.030504004395265606), (0.08640318897032012, 0.030675309542199148), (0.09036990980396087, 0.03191849333402471), (0.08807433400728253, 0.03290646726223996), (0.08866806731559626, 0.03728726453805084), (0.08543343575292071, 0.02718762541630789), (0.08370713516111211, 0.030215511135668078), (0.08456814843330018, 0.02546164231510412), (0.08555768005742968, 0.026243070552042298), (0.08406679695149158, 0.02438826052491033), (0.08695308402142944, 0.02926245130243095), (0.08093989759976632, 0.032639773078865474), (0.013364278458885503, 0.0011988283372310012), (0.10311750599520383, 0.007770672300249229), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None), (None, None)]), max_seq_len_source=513, max_seq_len_target=129, num_source_factors=1, num_target_factors=1, eop_id=-1), vocab_source_size=1232, vocab_target_size=5976, config_embed_source=EmbeddingConfig(vocab_size=1232, num_embed=512, dropout=0.0, num_factors=1, factor_configs=None, allow_sparse_grad=False), config_embed_target=EmbeddingConfig(vocab_size=5976, num_embed=512, dropout=0.0, num_factors=1, factor_configs=None, allow_sparse_grad=False), config_encoder=TransformerConfig(model_size=512, attention_heads=8, feed_forward_num_hidden=2048, act_type='relu', num_layers=6, dropout_attention=0.1, dropout_act=0.1, dropout_prepost=0.1, positional_embedding_type='fixed', preprocess_sequence='n', postprocess_sequence='dr', max_seq_len_source=513, max_seq_len_target=129, decoder_type='transformer', block_prepended_cross_attention=False, use_lhuc=False, depth_key_value=512, use_glu=False), config_decoder=TransformerConfig(model_size=512, attention_heads=8, feed_forward_num_hidden=2048, act_type='relu', num_layers=6, dropout_attention=0.1, dropout_act=0.1, dropout_prepost=0.1, positional_embedding_type='fixed', preprocess_sequence='n', postprocess_sequence='dr', max_seq_len_source=513, max_seq_len_target=129, decoder_type='transformer', block_prepended_cross_attention=False, use_lhuc=False, depth_key_value=512, use_glu=False), config_length_task=None, weight_tying_type='none', lhuc=False, dtype='float32', neural_vocab_selection=None, neural_vocab_selection_block_loss=False) [2023-12-17:11:19:29:INFO:sockeye.model:load_parameters] Loaded params from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded/model/params.best" to "cuda:0" [2023-12-17:11:19:29:INFO:sockeye.utils:log_parameters] # of parameters: 51188568 | trainable: 50859864 (99.36%) | shared: 3065688 (5.99%) | fixed: 328704 (0.64%) [2023-12-17:11:19:29:INFO:sockeye.utils:log_parameters] Trainable parameters: ['embedding_source.embedding [(1232, 512), float32]', 'embedding_target.embedding [(5976, 512), float32]', 'encoder.layers.0.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.0.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.0.self_attention.ff_out [(512, 512), float32]', 'encoder.layers.0.self_attention.ff_in [(1536, 512), float32]', 'encoder.layers.0.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.0.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.0.ff.ff1 [(2048, 512), float32]', 'encoder.layers.0.ff.ff1 [(2048,), float32]', 'encoder.layers.0.ff.ff2 [(512, 2048), float32]', 'encoder.layers.0.ff.ff2 [(512,), float32]', 'encoder.layers.1.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.1.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.1.self_attention.ff_out [(512, 512), float32]', 'encoder.layers.1.self_attention.ff_in [(1536, 512), float32]', 'encoder.layers.1.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.1.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.1.ff.ff1 [(2048, 512), float32]', 'encoder.layers.1.ff.ff1 [(2048,), float32]', 'encoder.layers.1.ff.ff2 [(512, 2048), float32]', 'encoder.layers.1.ff.ff2 [(512,), float32]', 'encoder.layers.2.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.2.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.2.self_attention.ff_out [(512, 512), float32]', 'encoder.layers.2.self_attention.ff_in [(1536, 512), float32]', 'encoder.layers.2.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.2.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.2.ff.ff1 [(2048, 512), float32]', 'encoder.layers.2.ff.ff1 [(2048,), float32]', 'encoder.layers.2.ff.ff2 [(512, 2048), float32]', 'encoder.layers.2.ff.ff2 [(512,), float32]', 'encoder.layers.3.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.3.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.3.self_attention.ff_out [(512, 512), float32]', 'encoder.layers.3.self_attention.ff_in [(1536, 512), float32]', 'encoder.layers.3.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.3.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.3.ff.ff1 [(2048, 512), float32]', 'encoder.layers.3.ff.ff1 [(2048,), float32]', 'encoder.layers.3.ff.ff2 [(512, 2048), float32]', 'encoder.layers.3.ff.ff2 [(512,), float32]', 'encoder.layers.4.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.4.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.4.self_attention.ff_out [(512, 512), float32]', 'encoder.layers.4.self_attention.ff_in [(1536, 512), float32]', 'encoder.layers.4.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.4.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.4.ff.ff1 [(2048, 512), float32]', 'encoder.layers.4.ff.ff1 [(2048,), float32]', 'encoder.layers.4.ff.ff2 [(512, 2048), float32]', 'encoder.layers.4.ff.ff2 [(512,), float32]', 'encoder.layers.5.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.5.pre_self_attention.layer_norm [(512,), float32]', 'encoder.layers.5.self_attention.ff_out [(512, 512), float32]', 'encoder.layers.5.self_attention.ff_in [(1536, 512), float32]', 'encoder.layers.5.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.5.pre_ff.layer_norm [(512,), float32]', 'encoder.layers.5.ff.ff1 [(2048, 512), float32]', 'encoder.layers.5.ff.ff1 [(2048,), float32]', 'encoder.layers.5.ff.ff2 [(512, 2048), float32]', 'encoder.layers.5.ff.ff2 [(512,), float32]', 'encoder.final_process.layer_norm [(512,), float32]', 'encoder.final_process.layer_norm [(512,), float32]', 'decoder.layers.0.autoregr_layer.ff_out [(512, 512), float32]', 'decoder.layers.0.autoregr_layer.ff_in [(1536, 512), float32]', 'decoder.layers.0.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.0.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.0.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.0.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.0.enc_attention.ff_out [(512, 512), float32]', 'decoder.layers.0.enc_attention.ff_q [(512, 512), float32]', 'decoder.layers.0.enc_attention.ff_kv [(1024, 512), float32]', 'decoder.layers.0.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.0.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.0.ff.ff1 [(2048, 512), float32]', 'decoder.layers.0.ff.ff1 [(2048,), float32]', 'decoder.layers.0.ff.ff2 [(512, 2048), float32]', 'decoder.layers.0.ff.ff2 [(512,), float32]', 'decoder.layers.1.autoregr_layer.ff_out [(512, 512), float32]', 'decoder.layers.1.autoregr_layer.ff_in [(1536, 512), float32]', 'decoder.layers.1.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.1.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.1.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.1.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.1.enc_attention.ff_out [(512, 512), float32]', 'decoder.layers.1.enc_attention.ff_q [(512, 512), float32]', 'decoder.layers.1.enc_attention.ff_kv [(1024, 512), float32]', 'decoder.layers.1.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.1.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.1.ff.ff1 [(2048, 512), float32]', 'decoder.layers.1.ff.ff1 [(2048,), float32]', 'decoder.layers.1.ff.ff2 [(512, 2048), float32]', 'decoder.layers.1.ff.ff2 [(512,), float32]', 'decoder.layers.2.autoregr_layer.ff_out [(512, 512), float32]', 'decoder.layers.2.autoregr_layer.ff_in [(1536, 512), float32]', 'decoder.layers.2.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.2.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.2.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.2.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.2.enc_attention.ff_out [(512, 512), float32]', 'decoder.layers.2.enc_attention.ff_q [(512, 512), float32]', 'decoder.layers.2.enc_attention.ff_kv [(1024, 512), float32]', 'decoder.layers.2.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.2.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.2.ff.ff1 [(2048, 512), float32]', 'decoder.layers.2.ff.ff1 [(2048,), float32]', 'decoder.layers.2.ff.ff2 [(512, 2048), float32]', 'decoder.layers.2.ff.ff2 [(512,), float32]', 'decoder.layers.3.autoregr_layer.ff_out [(512, 512), float32]', 'decoder.layers.3.autoregr_layer.ff_in [(1536, 512), float32]', 'decoder.layers.3.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.3.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.3.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.3.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.3.enc_attention.ff_out [(512, 512), float32]', 'decoder.layers.3.enc_attention.ff_q [(512, 512), float32]', 'decoder.layers.3.enc_attention.ff_kv [(1024, 512), float32]', 'decoder.layers.3.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.3.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.3.ff.ff1 [(2048, 512), float32]', 'decoder.layers.3.ff.ff1 [(2048,), float32]', 'decoder.layers.3.ff.ff2 [(512, 2048), float32]', 'decoder.layers.3.ff.ff2 [(512,), float32]', 'decoder.layers.4.autoregr_layer.ff_out [(512, 512), float32]', 'decoder.layers.4.autoregr_layer.ff_in [(1536, 512), float32]', 'decoder.layers.4.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.4.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.4.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.4.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.4.enc_attention.ff_out [(512, 512), float32]', 'decoder.layers.4.enc_attention.ff_q [(512, 512), float32]', 'decoder.layers.4.enc_attention.ff_kv [(1024, 512), float32]', 'decoder.layers.4.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.4.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.4.ff.ff1 [(2048, 512), float32]', 'decoder.layers.4.ff.ff1 [(2048,), float32]', 'decoder.layers.4.ff.ff2 [(512, 2048), float32]', 'decoder.layers.4.ff.ff2 [(512,), float32]', 'decoder.layers.5.autoregr_layer.ff_out [(512, 512), float32]', 'decoder.layers.5.autoregr_layer.ff_in [(1536, 512), float32]', 'decoder.layers.5.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.5.pre_autoregr_layer.layer_norm [(512,), float32]', 'decoder.layers.5.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.5.pre_enc_attention.layer_norm [(512,), float32]', 'decoder.layers.5.enc_attention.ff_out [(512, 512), float32]', 'decoder.layers.5.enc_attention.ff_q [(512, 512), float32]', 'decoder.layers.5.enc_attention.ff_kv [(1024, 512), float32]', 'decoder.layers.5.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.5.pre_ff.layer_norm [(512,), float32]', 'decoder.layers.5.ff.ff1 [(2048, 512), float32]', 'decoder.layers.5.ff.ff1 [(2048,), float32]', 'decoder.layers.5.ff.ff2 [(512, 2048), float32]', 'decoder.layers.5.ff.ff2 [(512,), float32]', 'decoder.final_process.layer_norm [(512,), float32]', 'decoder.final_process.layer_norm [(512,), float32]', 'output_layer [(5976, 512), float32]', 'output_layer [(5976,), float32]', 'output_layer_module_cached [(5976, 512), float32]', 'output_layer_module_cached [(5976,), float32]', 'output_layer_script_cached [(5976, 512), float32]', 'output_layer_script_cached [(5976,), float32]'] [2023-12-17:11:19:29:INFO:sockeye.utils:log_parameters] Shared parameters: ['output_layer.weight = output_layer_module_cached.weight = output_layer_script_cached.weight', 'output_layer.bias = output_layer_module_cached.bias = output_layer_script_cached.bias'] [2023-12-17:11:19:29:INFO:sockeye.utils:log_parameters] Fixed parameters: ['encoder.pos_embedding [(513, 512), float32]', 'decoder.pos_embedding [(129, 512), float32]'] [2023-12-17:11:19:29:INFO:sockeye.loss:__init__] Loss: cross-entropy | weight=1.00 | metric: perplexity (ppl) | output_name: 'logits' | label_name: 'target_label' [2023-12-17:11:19:29:WARNING:sockeye.optimizers:get_optimizer] Cannot import NVIDIA Apex optimizers (FusedAdam, FusedSGD). Consider installing Apex for faster GPU training: https://github.com/NVIDIA/apex [2023-12-17:11:19:29:INFO:sockeye.lr_scheduler:__init__] Will reduce the learning rate by a factor of 0.90 whenever the validation score doesn't improve 8 times. [2023-12-17:11:19:29:INFO:__main__:train] Tracing SockeyeModel on a validation batch [2023-12-17:11:19:33:INFO:sockeye.training:__init__] tensorboard not found. Consider 'pip install tensorboard' to log events to Tensorboard. [2023-12-17:11:19:33:INFO:sockeye.inference:__init__] Translator (1 model(s) beam_size=5 algorithm=BeamSearch, beam_search_stop=all max_input_length=512 nbest_size=1 ensemble_mode=None max_batch_size=16 dtype=torch.float32 skip_nvs=False nvs_thresh=0.5) [2023-12-17:11:19:33:INFO:sockeye.checkpoint_decoder:__init__] Created CheckpointDecoder(max_input_len=-1, beam_size=5, num_sentences=500) [2023-12-17:11:20:03:INFO:sockeye.training:fit] Early stopping by optimizing 'chrf' [2023-12-17:11:20:03:INFO:sockeye.model:save_config] Saved model config to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/config" [2023-12-17:11:20:03:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00000" [2023-12-17:11:20:03:INFO:sockeye.training:fit] Training started. [2023-12-17:11:21:06:INFO:sockeye.training:__call__] E=0 B=50 s/sec=74.49 tok/sec=3434.09 u/sec=0.81 ppl=18.655668 [2023-12-17:11:21:08:INFO:sockeye.training:__call__] E=0 B=100 s/sec=2336.30 tok/sec=104114.57 u/sec=24.22 ppl=18.608390 [2023-12-17:11:21:10:INFO:sockeye.training:__call__] E=0 B=150 s/sec=2143.73 tok/sec=105962.53 u/sec=24.67 ppl=18.206416 [2023-12-17:11:21:12:INFO:sockeye.training:__call__] E=0 B=200 s/sec=2353.40 tok/sec=106096.56 u/sec=25.19 ppl=18.171534 [2023-12-17:11:21:14:INFO:sockeye.training:__call__] E=0 B=250 s/sec=2212.65 tok/sec=108159.94 u/sec=24.61 ppl=18.142329 [2023-12-17:11:21:16:INFO:sockeye.training:__call__] E=0 B=300 s/sec=2669.99 tok/sec=102130.10 u/sec=25.87 ppl=18.145817 [2023-12-17:11:21:18:INFO:sockeye.training:__call__] E=0 B=350 s/sec=1913.96 tok/sec=113237.37 u/sec=24.51 ppl=18.044639 [2023-12-17:11:21:20:INFO:sockeye.training:__call__] E=0 B=400 s/sec=2043.56 tok/sec=109010.56 u/sec=24.37 ppl=17.931931 [2023-12-17:11:21:22:INFO:sockeye.training:__call__] E=0 B=450 s/sec=2223.97 tok/sec=110042.76 u/sec=24.73 ppl=17.854408 [2023-12-17:11:21:24:INFO:sockeye.training:__call__] E=0 B=500 s/sec=2338.26 tok/sec=106896.46 u/sec=25.59 ppl=17.848924 [2023-12-17:11:21:26:INFO:sockeye.training:__call__] E=0 B=550 s/sec=2256.38 tok/sec=107088.11 u/sec=25.05 ppl=17.821833 [2023-12-17:11:21:28:INFO:sockeye.training:__call__] E=0 B=600 s/sec=2287.93 tok/sec=105791.94 u/sec=24.78 ppl=17.818209 [2023-12-17:11:21:30:INFO:sockeye.training:__call__] E=0 B=650 s/sec=2003.44 tok/sec=108956.76 u/sec=23.85 ppl=17.794558 [2023-12-17:11:21:32:INFO:sockeye.training:__call__] E=0 B=700 s/sec=1977.20 tok/sec=109670.89 u/sec=24.91 ppl=17.729489 [2023-12-17:11:21:34:INFO:sockeye.training:__call__] E=0 B=750 s/sec=1917.73 tok/sec=112075.36 u/sec=24.07 ppl=17.731481 [2023-12-17:11:21:36:INFO:sockeye.training:__call__] E=0 B=800 s/sec=2322.90 tok/sec=107205.81 u/sec=25.03 ppl=17.737991 [2023-12-17:11:21:38:INFO:sockeye.training:__call__] E=0 B=850 s/sec=2243.45 tok/sec=108070.36 u/sec=25.04 ppl=17.691493 [2023-12-17:11:21:40:INFO:sockeye.training:__call__] E=0 B=900 s/sec=2141.74 tok/sec=108902.87 u/sec=23.95 ppl=17.670818 [2023-12-17:11:21:42:INFO:sockeye.training:__call__] E=0 B=950 s/sec=2143.04 tok/sec=105702.48 u/sec=24.62 ppl=17.615685 [2023-12-17:11:21:44:INFO:sockeye.training:__call__] E=0 B=1000 s/sec=2139.96 tok/sec=108931.79 u/sec=24.77 ppl=17.619012 [2023-12-17:11:21:46:INFO:sockeye.training:__call__] E=0 B=1050 s/sec=2140.44 tok/sec=107687.76 u/sec=24.77 ppl=17.588894 [2023-12-17:11:21:48:INFO:sockeye.training:__call__] E=0 B=1100 s/sec=2443.49 tok/sec=105328.87 u/sec=25.67 ppl=17.617007 [2023-12-17:11:21:51:INFO:sockeye.training:__call__] E=0 B=1150 s/sec=1705.10 tok/sec=102555.01 u/sec=19.45 ppl=17.577034 [2023-12-17:11:21:53:INFO:sockeye.training:__call__] E=0 B=1200 s/sec=2241.08 tok/sec=106864.79 u/sec=25.10 ppl=17.573240 [2023-12-17:11:21:55:INFO:sockeye.training:__call__] E=0 B=1250 s/sec=2339.28 tok/sec=102717.32 u/sec=25.34 ppl=17.568516 [2023-12-17:11:21:57:INFO:sockeye.training:__call__] E=0 B=1300 s/sec=2094.58 tok/sec=110167.94 u/sec=24.11 ppl=17.560434 [2023-12-17:11:21:59:INFO:sockeye.training:__call__] E=0 B=1350 s/sec=2302.40 tok/sec=106618.90 u/sec=25.16 ppl=17.541674 [2023-12-17:11:22:01:INFO:sockeye.training:__call__] E=0 B=1400 s/sec=2500.07 tok/sec=99702.29 u/sec=26.09 ppl=17.561718 [2023-12-17:11:22:03:INFO:sockeye.training:__call__] E=0 B=1450 s/sec=2197.94 tok/sec=105695.57 u/sec=25.30 ppl=17.544423 [2023-12-17:11:22:04:INFO:sockeye.training:__call__] E=0 B=1500 s/sec=2291.61 tok/sec=105721.71 u/sec=25.30 ppl=17.518426 [2023-12-17:11:22:06:INFO:sockeye.training:__call__] E=0 B=1550 s/sec=2311.30 tok/sec=106807.95 u/sec=25.43 ppl=17.509666 [2023-12-17:11:22:08:INFO:sockeye.training:__call__] E=0 B=1600 s/sec=2014.89 tok/sec=111298.68 u/sec=24.99 ppl=17.483595 [2023-12-17:11:22:10:INFO:sockeye.training:__call__] E=0 B=1650 s/sec=2036.85 tok/sec=107306.52 u/sec=25.26 ppl=17.450279 [2023-12-17:11:22:12:INFO:sockeye.training:__call__] E=0 B=1700 s/sec=2217.83 tok/sec=107323.71 u/sec=25.11 ppl=17.437040 [2023-12-17:11:22:14:INFO:sockeye.training:__call__] E=0 B=1750 s/sec=2381.45 tok/sec=105723.47 u/sec=25.31 ppl=17.438608 [2023-12-17:11:22:16:INFO:sockeye.training:__call__] E=0 B=1800 s/sec=2232.63 tok/sec=108222.69 u/sec=25.01 ppl=17.423068 [2023-12-17:11:22:18:INFO:sockeye.training:__call__] E=0 B=1850 s/sec=2419.68 tok/sec=104562.83 u/sec=25.63 ppl=17.400424 [2023-12-17:11:22:20:INFO:sockeye.training:__call__] E=0 B=1900 s/sec=2112.82 tok/sec=108239.43 u/sec=24.68 ppl=17.392888 [2023-12-17:11:22:22:INFO:sockeye.training:__call__] E=0 B=1950 s/sec=2145.62 tok/sec=108062.43 u/sec=24.93 ppl=17.394044 [2023-12-17:11:22:24:INFO:sockeye.training:__call__] E=0 B=2000 s/sec=2343.37 tok/sec=101339.48 u/sec=25.83 ppl=17.364784 [2023-12-17:11:22:26:INFO:sockeye.training:__call__] E=0 B=2050 s/sec=2532.08 tok/sec=103286.03 u/sec=25.65 ppl=17.370202 [2023-12-17:11:22:28:INFO:sockeye.training:__call__] E=0 B=2100 s/sec=2451.83 tok/sec=104512.98 u/sec=25.67 ppl=17.363843 [2023-12-17:11:22:30:INFO:sockeye.training:__call__] E=0 B=2150 s/sec=2316.15 tok/sec=106574.82 u/sec=24.79 ppl=17.346560 [2023-12-17:11:22:32:INFO:sockeye.training:__call__] E=0 B=2200 s/sec=2413.29 tok/sec=103641.06 u/sec=25.48 ppl=17.336061 [2023-12-17:11:22:34:INFO:sockeye.training:__call__] E=0 B=2250 s/sec=2295.75 tok/sec=105572.35 u/sec=25.26 ppl=17.321225 [2023-12-17:11:22:36:INFO:sockeye.training:__call__] E=0 B=2300 s/sec=2298.07 tok/sec=106597.76 u/sec=25.42 ppl=17.332496 [2023-12-17:11:22:38:INFO:sockeye.training:__call__] E=0 B=2350 s/sec=2143.55 tok/sec=108318.89 u/sec=24.95 ppl=17.326014 [2023-12-17:11:22:40:INFO:sockeye.training:__call__] E=0 B=2400 s/sec=2281.87 tok/sec=103521.53 u/sec=25.65 ppl=17.303162 [2023-12-17:11:22:42:INFO:sockeye.training:__call__] E=0 B=2450 s/sec=2369.40 tok/sec=107841.93 u/sec=25.14 ppl=17.311880 [2023-12-17:11:22:44:INFO:sockeye.training:__call__] E=0 B=2500 s/sec=2322.30 tok/sec=105678.93 u/sec=25.42 ppl=17.309172 [2023-12-17:11:22:46:INFO:sockeye.training:__call__] E=0 B=2550 s/sec=2241.69 tok/sec=108404.40 u/sec=25.11 ppl=17.305746 [2023-12-17:11:22:48:INFO:sockeye.training:__call__] E=0 B=2600 s/sec=2369.07 tok/sec=105001.36 u/sec=25.89 ppl=17.295458 [2023-12-17:11:22:50:INFO:sockeye.training:__call__] E=0 B=2650 s/sec=2157.39 tok/sec=109138.61 u/sec=24.79 ppl=17.283534 [2023-12-17:11:22:52:INFO:sockeye.training:__call__] E=0 B=2700 s/sec=2333.57 tok/sec=107230.36 u/sec=25.41 ppl=17.280709 [2023-12-17:11:22:54:INFO:sockeye.training:__call__] E=0 B=2750 s/sec=2552.08 tok/sec=101848.12 u/sec=26.23 ppl=17.274363 [2023-12-17:11:22:56:INFO:sockeye.training:__call__] E=0 B=2800 s/sec=1847.99 tok/sec=114121.98 u/sec=24.01 ppl=17.267134 [2023-12-17:11:22:58:INFO:sockeye.training:__call__] E=0 B=2850 s/sec=2237.11 tok/sec=107030.57 u/sec=24.92 ppl=17.267797 [2023-12-17:11:23:00:INFO:sockeye.training:__call__] E=0 B=2900 s/sec=2101.49 tok/sec=106788.73 u/sec=24.46 ppl=17.250172 [2023-12-17:11:23:02:INFO:sockeye.training:__call__] E=0 B=2950 s/sec=2104.02 tok/sec=109694.65 u/sec=24.81 ppl=17.241270 [2023-12-17:11:23:04:INFO:sockeye.training:__call__] E=0 B=3000 s/sec=2358.34 tok/sec=103750.13 u/sec=26.00 ppl=17.233959 [2023-12-17:11:23:06:INFO:sockeye.training:__call__] E=0 B=3050 s/sec=2068.26 tok/sec=109135.07 u/sec=24.53 ppl=17.230318 [2023-12-17:11:23:08:INFO:sockeye.training:__call__] E=0 B=3100 s/sec=2576.00 tok/sec=102635.79 u/sec=26.31 ppl=17.234948 [2023-12-17:11:23:10:INFO:sockeye.training:__call__] E=0 B=3150 s/sec=1994.04 tok/sec=110714.60 u/sec=24.78 ppl=17.233230 [2023-12-17:11:23:12:INFO:sockeye.training:__call__] E=0 B=3200 s/sec=1958.45 tok/sec=108967.01 u/sec=25.08 ppl=17.212959 [2023-12-17:11:23:14:INFO:sockeye.training:__call__] E=0 B=3250 s/sec=2463.45 tok/sec=104839.28 u/sec=25.12 ppl=17.213881 [2023-12-17:11:23:16:INFO:sockeye.training:__call__] E=0 B=3300 s/sec=2195.44 tok/sec=108452.42 u/sec=24.99 ppl=17.212524 [2023-12-17:11:23:18:INFO:sockeye.training:__call__] E=0 B=3350 s/sec=2145.07 tok/sec=108704.53 u/sec=24.78 ppl=17.206518 [2023-12-17:11:23:20:INFO:sockeye.training:__call__] E=0 B=3400 s/sec=2041.84 tok/sec=113002.81 u/sec=24.35 ppl=17.200257 [2023-12-17:11:23:22:INFO:sockeye.training:__call__] E=0 B=3450 s/sec=2266.35 tok/sec=107219.52 u/sec=25.48 ppl=17.189116 [2023-12-17:11:23:24:INFO:sockeye.training:__call__] E=0 B=3500 s/sec=2234.35 tok/sec=107509.51 u/sec=25.44 ppl=17.176347 [2023-12-17:11:23:26:INFO:sockeye.training:__call__] E=0 B=3550 s/sec=2147.71 tok/sec=105614.64 u/sec=25.62 ppl=17.161121 [2023-12-17:11:23:28:INFO:sockeye.training:__call__] E=0 B=3600 s/sec=2065.38 tok/sec=108923.99 u/sec=25.07 ppl=17.152146 [2023-12-17:11:23:30:INFO:sockeye.training:__call__] E=0 B=3650 s/sec=2364.99 tok/sec=107029.80 u/sec=25.22 ppl=17.141910 [2023-12-17:11:23:32:INFO:sockeye.training:__call__] E=0 B=3700 s/sec=2103.91 tok/sec=108482.59 u/sec=25.24 ppl=17.141180 [2023-12-17:11:23:34:INFO:sockeye.training:__call__] E=0 B=3750 s/sec=2154.56 tok/sec=107361.53 u/sec=25.46 ppl=17.122229 [2023-12-17:11:23:36:INFO:sockeye.training:__call__] E=0 B=3800 s/sec=2253.65 tok/sec=107259.04 u/sec=25.56 ppl=17.113981 [2023-12-17:11:23:38:INFO:sockeye.training:__call__] E=0 B=3850 s/sec=2332.09 tok/sec=106310.37 u/sec=25.57 ppl=17.113770 [2023-12-17:11:23:40:INFO:sockeye.training:__call__] E=0 B=3900 s/sec=2314.67 tok/sec=108216.83 u/sec=24.52 ppl=17.106544 [2023-12-17:11:23:42:INFO:sockeye.training:__call__] E=0 B=3950 s/sec=2416.28 tok/sec=104743.67 u/sec=25.60 ppl=17.104703 [2023-12-17:11:23:44:INFO:sockeye.training:__call__] E=1 B=4000 s/sec=2193.01 tok/sec=100832.62 u/sec=23.88 ppl=17.100023 [2023-12-17:11:23:44:INFO:sockeye.training:_create_checkpoint] Checkpoint [1] Updates=4000 Epoch=1 Samples=356496 Time-cost=220.406 Updates/sec=18.148 [2023-12-17:11:23:44:INFO:sockeye.training:_create_checkpoint] Checkpoint [1] Train-ppl=17.100023 [2023-12-17:11:25:05:INFO:sockeye.training:_evaluate] Checkpoint [1] Validation-ppl=22.523855 Validation-bleu=0.127874 Validation-chrf=28.768187 Validation-rouge1=0.157579 Validation-rouge2=0.074995 Validation-rougel=0.151230 Validation-length-ratio-mse=1.283504 Validation-ter=97.234833 Validation-avg-sec-per-sent=0.160033 Validation-decode-walltime=80.016319 [2023-12-17:11:25:05:INFO:sockeye.training:_determine_improvement] Validation-chrf improved to 28.768187 (delta=28.768187). [2023-12-17:11:25:05:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [1] Learning-rate=0.000200 [2023-12-17:11:25:05:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:25:05:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:25:06:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00001" [2023-12-17:11:25:06:INFO:sockeye.training:_update_best_params] '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best' now points to 'params.00001' [2023-12-17:11:25:06:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl" [2023-12-17:11:25:06:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/lr_scheduler_best.pkl' [2023-12-17:11:25:08:INFO:sockeye.training:__call__] E=1 B=4050 s/sec=55.36 tok/sec=2426.19 u/sec=0.59 ppl=12.546023 [2023-12-17:11:25:10:INFO:sockeye.training:__call__] E=1 B=4100 s/sec=2266.37 tok/sec=107137.49 u/sec=24.76 ppl=12.526253 [2023-12-17:11:25:12:INFO:sockeye.training:__call__] E=1 B=4150 s/sec=2506.03 tok/sec=104094.26 u/sec=25.51 ppl=12.690548 [2023-12-17:11:25:14:INFO:sockeye.training:__call__] E=1 B=4200 s/sec=2000.83 tok/sec=109130.50 u/sec=25.31 ppl=12.582174 [2023-12-17:11:25:16:INFO:sockeye.training:__call__] E=1 B=4250 s/sec=2383.59 tok/sec=105350.53 u/sec=25.47 ppl=12.589332 [2023-12-17:11:25:18:INFO:sockeye.training:__call__] E=1 B=4300 s/sec=1981.98 tok/sec=111549.48 u/sec=24.24 ppl=12.600232 [2023-12-17:11:25:20:INFO:sockeye.training:__call__] E=1 B=4350 s/sec=2434.26 tok/sec=105757.29 u/sec=24.82 ppl=12.679638 [2023-12-17:11:25:22:INFO:sockeye.training:__call__] E=1 B=4400 s/sec=2090.83 tok/sec=107477.75 u/sec=25.28 ppl=12.692391 [2023-12-17:11:25:24:INFO:sockeye.training:__call__] E=1 B=4450 s/sec=2255.01 tok/sec=106352.03 u/sec=25.44 ppl=12.715244 [2023-12-17:11:25:26:INFO:sockeye.training:__call__] E=1 B=4500 s/sec=2468.43 tok/sec=104014.05 u/sec=25.29 ppl=12.760256 [2023-12-17:11:25:28:INFO:sockeye.training:__call__] E=1 B=4550 s/sec=2240.17 tok/sec=108569.51 u/sec=24.65 ppl=12.793539 [2023-12-17:11:25:30:INFO:sockeye.training:__call__] E=1 B=4600 s/sec=2031.87 tok/sec=107564.89 u/sec=25.05 ppl=12.783694 [2023-12-17:11:25:32:INFO:sockeye.training:__call__] E=1 B=4650 s/sec=2187.24 tok/sec=106870.18 u/sec=24.11 ppl=12.774624 [2023-12-17:11:25:34:INFO:sockeye.training:__call__] E=1 B=4700 s/sec=2399.69 tok/sec=105406.31 u/sec=25.55 ppl=12.777491 [2023-12-17:11:25:36:INFO:sockeye.training:__call__] E=1 B=4750 s/sec=2397.24 tok/sec=104377.03 u/sec=25.31 ppl=12.822910 [2023-12-17:11:25:38:INFO:sockeye.training:__call__] E=1 B=4800 s/sec=2161.12 tok/sec=110515.69 u/sec=24.87 ppl=12.831135 [2023-12-17:11:25:40:INFO:sockeye.training:__call__] E=1 B=4850 s/sec=2090.30 tok/sec=110944.35 u/sec=24.74 ppl=12.850528 [2023-12-17:11:25:42:INFO:sockeye.training:__call__] E=1 B=4900 s/sec=2312.34 tok/sec=107075.96 u/sec=24.87 ppl=12.868678 [2023-12-17:11:25:44:INFO:sockeye.training:__call__] E=1 B=4950 s/sec=2498.47 tok/sec=103871.53 u/sec=25.39 ppl=12.926488 [2023-12-17:11:25:46:INFO:sockeye.training:__call__] E=1 B=5000 s/sec=2368.04 tok/sec=108084.60 u/sec=24.92 ppl=12.973971 [2023-12-17:11:25:48:INFO:sockeye.training:__call__] E=1 B=5050 s/sec=1941.29 tok/sec=106541.83 u/sec=23.88 ppl=13.002768 [2023-12-17:11:25:50:INFO:sockeye.training:__call__] E=1 B=5100 s/sec=2188.98 tok/sec=106450.36 u/sec=25.06 ppl=13.029866 [2023-12-17:11:25:52:INFO:sockeye.training:__call__] E=1 B=5150 s/sec=2350.78 tok/sec=104982.85 u/sec=24.94 ppl=13.058025 [2023-12-17:11:25:54:INFO:sockeye.training:__call__] E=1 B=5200 s/sec=2192.34 tok/sec=107880.69 u/sec=25.23 ppl=13.076669 [2023-12-17:11:25:56:INFO:sockeye.training:__call__] E=1 B=5250 s/sec=2252.10 tok/sec=109027.27 u/sec=24.96 ppl=13.099099 [2023-12-17:11:25:58:INFO:sockeye.training:__call__] E=1 B=5300 s/sec=2126.77 tok/sec=108129.42 u/sec=24.99 ppl=13.110970 [2023-12-17:11:26:00:INFO:sockeye.training:__call__] E=1 B=5350 s/sec=2069.00 tok/sec=111410.09 u/sec=24.68 ppl=13.117524 [2023-12-17:11:26:02:INFO:sockeye.training:__call__] E=1 B=5400 s/sec=2106.33 tok/sec=116710.60 u/sec=23.14 ppl=13.126858 [2023-12-17:11:26:04:INFO:sockeye.training:__call__] E=1 B=5450 s/sec=2435.14 tok/sec=103689.82 u/sec=25.84 ppl=13.149983 [2023-12-17:11:26:06:INFO:sockeye.training:__call__] E=1 B=5500 s/sec=2064.90 tok/sec=108371.03 u/sec=25.11 ppl=13.160420 [2023-12-17:11:26:08:INFO:sockeye.training:__call__] E=1 B=5550 s/sec=2182.28 tok/sec=108247.18 u/sec=24.62 ppl=13.178994 [2023-12-17:11:26:10:INFO:sockeye.training:__call__] E=1 B=5600 s/sec=2136.41 tok/sec=108412.98 u/sec=24.64 ppl=13.193953 [2023-12-17:11:26:12:INFO:sockeye.training:__call__] E=1 B=5650 s/sec=2160.95 tok/sec=106333.04 u/sec=25.15 ppl=13.193265 [2023-12-17:11:26:14:INFO:sockeye.training:__call__] E=1 B=5700 s/sec=2227.10 tok/sec=98617.36 u/sec=24.42 ppl=13.201182 [2023-12-17:11:26:16:INFO:sockeye.training:__call__] E=1 B=5750 s/sec=2298.46 tok/sec=104373.14 u/sec=25.84 ppl=13.201866 [2023-12-17:11:26:18:INFO:sockeye.training:__call__] E=1 B=5800 s/sec=2031.43 tok/sec=107786.04 u/sec=25.09 ppl=13.205235 [2023-12-17:11:26:20:INFO:sockeye.training:__call__] E=1 B=5850 s/sec=2468.43 tok/sec=101243.65 u/sec=26.46 ppl=13.217665 [2023-12-17:11:26:22:INFO:sockeye.training:__call__] E=1 B=5900 s/sec=2571.65 tok/sec=104561.32 u/sec=25.47 ppl=13.231522 [2023-12-17:11:26:24:INFO:sockeye.training:__call__] E=1 B=5950 s/sec=2325.27 tok/sec=106760.12 u/sec=25.27 ppl=13.241720 [2023-12-17:11:26:26:INFO:sockeye.training:__call__] E=1 B=6000 s/sec=1966.04 tok/sec=109630.20 u/sec=24.38 ppl=13.256885 [2023-12-17:11:26:28:INFO:sockeye.training:__call__] E=1 B=6050 s/sec=2528.83 tok/sec=103801.99 u/sec=25.87 ppl=13.280794 [2023-12-17:11:26:30:INFO:sockeye.training:__call__] E=1 B=6100 s/sec=2206.18 tok/sec=105671.38 u/sec=25.16 ppl=13.286274 [2023-12-17:11:26:32:INFO:sockeye.training:__call__] E=1 B=6150 s/sec=2417.30 tok/sec=105099.89 u/sec=25.26 ppl=13.304885 [2023-12-17:11:26:34:INFO:sockeye.training:__call__] E=1 B=6200 s/sec=2152.95 tok/sec=108124.27 u/sec=25.25 ppl=13.316943 [2023-12-17:11:26:36:INFO:sockeye.training:__call__] E=1 B=6250 s/sec=2499.69 tok/sec=104216.61 u/sec=25.61 ppl=13.333610 [2023-12-17:11:26:38:INFO:sockeye.training:__call__] E=1 B=6300 s/sec=2186.63 tok/sec=109834.86 u/sec=24.58 ppl=13.349518 [2023-12-17:11:26:40:INFO:sockeye.training:__call__] E=1 B=6350 s/sec=2240.49 tok/sec=108143.90 u/sec=24.70 ppl=13.362637 [2023-12-17:11:26:42:INFO:sockeye.training:__call__] E=1 B=6400 s/sec=2118.71 tok/sec=108649.10 u/sec=24.80 ppl=13.372965 [2023-12-17:11:26:44:INFO:sockeye.training:__call__] E=1 B=6450 s/sec=2323.65 tok/sec=105552.31 u/sec=25.00 ppl=13.377158 [2023-12-17:11:26:46:INFO:sockeye.training:__call__] E=1 B=6500 s/sec=1970.17 tok/sec=109496.44 u/sec=25.03 ppl=13.375593 [2023-12-17:11:26:48:INFO:sockeye.training:__call__] E=1 B=6550 s/sec=2559.75 tok/sec=103018.36 u/sec=25.85 ppl=13.393016 [2023-12-17:11:26:50:INFO:sockeye.training:__call__] E=1 B=6600 s/sec=2258.82 tok/sec=104926.05 u/sec=25.30 ppl=13.406176 [2023-12-17:11:26:52:INFO:sockeye.training:__call__] E=1 B=6650 s/sec=2282.19 tok/sec=104792.43 u/sec=25.61 ppl=13.415947 [2023-12-17:11:26:54:INFO:sockeye.training:__call__] E=1 B=6700 s/sec=2275.23 tok/sec=103977.39 u/sec=25.71 ppl=13.436465 [2023-12-17:11:26:56:INFO:sockeye.training:__call__] E=1 B=6750 s/sec=2274.26 tok/sec=106764.39 u/sec=25.34 ppl=13.445209 [2023-12-17:11:26:58:INFO:sockeye.training:__call__] E=1 B=6800 s/sec=2261.88 tok/sec=107134.01 u/sec=24.93 ppl=13.447806 [2023-12-17:11:27:00:INFO:sockeye.training:__call__] E=1 B=6850 s/sec=2086.05 tok/sec=106348.51 u/sec=25.17 ppl=13.455241 [2023-12-17:11:27:02:INFO:sockeye.training:__call__] E=1 B=6900 s/sec=2246.42 tok/sec=109863.91 u/sec=24.85 ppl=13.467257 [2023-12-17:11:27:04:INFO:sockeye.training:__call__] E=1 B=6950 s/sec=2436.83 tok/sec=105674.67 u/sec=25.43 ppl=13.479890 [2023-12-17:11:27:06:INFO:sockeye.training:__call__] E=1 B=7000 s/sec=1994.80 tok/sec=110562.08 u/sec=24.21 ppl=13.482153 [2023-12-17:11:27:08:INFO:sockeye.training:__call__] E=1 B=7050 s/sec=2147.21 tok/sec=106198.05 u/sec=24.71 ppl=13.488930 [2023-12-17:11:27:10:INFO:sockeye.training:__call__] E=1 B=7100 s/sec=2259.36 tok/sec=105939.81 u/sec=24.82 ppl=13.502704 [2023-12-17:11:27:12:INFO:sockeye.training:__call__] E=1 B=7150 s/sec=2085.02 tok/sec=108800.35 u/sec=24.68 ppl=13.515030 [2023-12-17:11:27:14:INFO:sockeye.training:__call__] E=1 B=7200 s/sec=2266.29 tok/sec=106920.82 u/sec=25.07 ppl=13.520870 [2023-12-17:11:27:16:INFO:sockeye.training:__call__] E=1 B=7250 s/sec=2286.53 tok/sec=106683.70 u/sec=24.72 ppl=13.538700 [2023-12-17:11:27:18:INFO:sockeye.training:__call__] E=1 B=7300 s/sec=2118.74 tok/sec=107977.32 u/sec=24.52 ppl=13.547332 [2023-12-17:11:27:20:INFO:sockeye.training:__call__] E=1 B=7350 s/sec=2337.78 tok/sec=106107.89 u/sec=25.11 ppl=13.551338 [2023-12-17:11:27:22:INFO:sockeye.training:__call__] E=1 B=7400 s/sec=2264.51 tok/sec=105342.72 u/sec=25.69 ppl=13.563710 [2023-12-17:11:27:24:INFO:sockeye.training:__call__] E=1 B=7450 s/sec=2442.06 tok/sec=103292.74 u/sec=26.09 ppl=13.571934 [2023-12-17:11:27:26:INFO:sockeye.training:__call__] E=1 B=7500 s/sec=2059.74 tok/sec=108437.43 u/sec=24.76 ppl=13.583101 [2023-12-17:11:27:28:INFO:sockeye.training:__call__] E=1 B=7550 s/sec=2215.62 tok/sec=106970.18 u/sec=25.50 ppl=13.592012 [2023-12-17:11:27:30:INFO:sockeye.training:__call__] E=1 B=7600 s/sec=2018.45 tok/sec=111973.25 u/sec=24.35 ppl=13.599093 [2023-12-17:11:27:32:INFO:sockeye.training:__call__] E=1 B=7650 s/sec=1959.23 tok/sec=110060.23 u/sec=24.39 ppl=13.609347 [2023-12-17:11:27:34:INFO:sockeye.training:__call__] E=1 B=7700 s/sec=2165.70 tok/sec=108624.43 u/sec=24.66 ppl=13.616538 [2023-12-17:11:27:36:INFO:sockeye.training:__call__] E=1 B=7750 s/sec=2358.10 tok/sec=103258.20 u/sec=25.11 ppl=13.623370 [2023-12-17:11:27:38:INFO:sockeye.training:__call__] E=1 B=7800 s/sec=1930.17 tok/sec=113779.90 u/sec=24.03 ppl=13.625021 [2023-12-17:11:27:40:INFO:sockeye.training:__call__] E=1 B=7850 s/sec=2391.17 tok/sec=105260.55 u/sec=25.63 ppl=13.633094 [2023-12-17:11:27:42:INFO:sockeye.training:__call__] E=1 B=7900 s/sec=2442.33 tok/sec=103266.13 u/sec=25.92 ppl=13.641049 [2023-12-17:11:27:44:INFO:sockeye.training:__call__] E=1 B=7950 s/sec=1936.55 tok/sec=111458.24 u/sec=24.26 ppl=13.646912 [2023-12-17:11:27:46:INFO:sockeye.training:__call__] E=2 B=8000 s/sec=1856.21 tok/sec=103831.14 u/sec=22.75 ppl=13.640558 [2023-12-17:11:27:46:INFO:sockeye.training:_create_checkpoint] Checkpoint [2] Updates=8000 Epoch=2 Samples=712944 Time-cost=160.105 Updates/sec=24.984 [2023-12-17:11:27:46:INFO:sockeye.training:_create_checkpoint] Checkpoint [2] Train-ppl=13.640558 [2023-12-17:11:27:49:INFO:sockeye.training:_evaluate] Checkpoint [2] Validation-ppl=23.475334 Validation-bleu=0.151654 Validation-chrf=30.600195 Validation-rouge1=0.160526 Validation-rouge2=0.070579 Validation-rougel=0.152247 Validation-length-ratio-mse=1.298197 Validation-ter=96.822121 Validation-avg-sec-per-sent=0.003334 Validation-decode-walltime=1.667208 [2023-12-17:11:27:49:INFO:sockeye.training:_determine_improvement] Validation-chrf improved to 30.600195 (delta=1.832007). [2023-12-17:11:27:49:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [2] Learning-rate=0.000200 [2023-12-17:11:27:49:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:27:49:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:27:50:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00002" [2023-12-17:11:27:50:INFO:sockeye.training:_update_best_params] '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best' now points to 'params.00002' [2023-12-17:11:27:50:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl" [2023-12-17:11:27:50:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/lr_scheduler_best.pkl' [2023-12-17:11:27:52:INFO:sockeye.training:__call__] E=2 B=8050 s/sec=731.32 tok/sec=34585.05 u/sec=8.43 ppl=10.624878 [2023-12-17:11:27:54:INFO:sockeye.training:__call__] E=2 B=8100 s/sec=2057.05 tok/sec=106232.86 u/sec=24.92 ppl=10.683971 [2023-12-17:11:27:56:INFO:sockeye.training:__call__] E=2 B=8150 s/sec=2110.15 tok/sec=108064.00 u/sec=23.85 ppl=10.698346 [2023-12-17:11:27:58:INFO:sockeye.training:__call__] E=2 B=8200 s/sec=2116.70 tok/sec=107583.65 u/sec=24.73 ppl=10.732954 [2023-12-17:11:28:00:INFO:sockeye.training:__call__] E=2 B=8250 s/sec=2063.84 tok/sec=106158.31 u/sec=24.52 ppl=10.668729 [2023-12-17:11:28:02:INFO:sockeye.training:__call__] E=2 B=8300 s/sec=2365.75 tok/sec=97638.56 u/sec=25.28 ppl=10.723327 [2023-12-17:11:28:04:INFO:sockeye.training:__call__] E=2 B=8350 s/sec=2321.86 tok/sec=105780.51 u/sec=25.02 ppl=10.802803 [2023-12-17:11:28:06:INFO:sockeye.training:__call__] E=2 B=8400 s/sec=2380.80 tok/sec=106433.92 u/sec=25.22 ppl=10.840915 [2023-12-17:11:28:08:INFO:sockeye.training:__call__] E=2 B=8450 s/sec=2452.00 tok/sec=105386.56 u/sec=25.71 ppl=10.882255 [2023-12-17:11:28:10:INFO:sockeye.training:__call__] E=2 B=8500 s/sec=2098.47 tok/sec=105825.20 u/sec=24.29 ppl=10.880357 [2023-12-17:11:28:12:INFO:sockeye.training:__call__] E=2 B=8550 s/sec=2111.18 tok/sec=110815.85 u/sec=24.12 ppl=10.937135 [2023-12-17:11:28:14:INFO:sockeye.training:__call__] E=2 B=8600 s/sec=2064.92 tok/sec=111741.13 u/sec=24.54 ppl=10.945506 [2023-12-17:11:28:16:INFO:sockeye.training:__call__] E=2 B=8650 s/sec=2151.12 tok/sec=107763.80 u/sec=24.99 ppl=10.957841 [2023-12-17:11:28:18:INFO:sockeye.training:__call__] E=2 B=8700 s/sec=2116.04 tok/sec=109841.47 u/sec=25.05 ppl=10.995890 [2023-12-17:11:28:20:INFO:sockeye.training:__call__] E=2 B=8750 s/sec=1988.00 tok/sec=108127.93 u/sec=24.75 ppl=11.008866 [2023-12-17:11:28:22:INFO:sockeye.training:__call__] E=2 B=8800 s/sec=2227.56 tok/sec=109711.15 u/sec=25.04 ppl=11.021114 [2023-12-17:11:28:24:INFO:sockeye.training:__call__] E=2 B=8850 s/sec=2085.81 tok/sec=107573.98 u/sec=24.78 ppl=11.017539 [2023-12-17:11:28:26:INFO:sockeye.training:__call__] E=2 B=8900 s/sec=2276.92 tok/sec=101588.25 u/sec=24.37 ppl=11.061331 [2023-12-17:11:28:28:INFO:sockeye.training:__call__] E=2 B=8950 s/sec=2016.83 tok/sec=103298.94 u/sec=24.15 ppl=11.068868 [2023-12-17:11:28:31:INFO:sockeye.training:__call__] E=2 B=9000 s/sec=1810.27 tok/sec=119595.64 u/sec=23.00 ppl=11.079225 [2023-12-17:11:28:33:INFO:sockeye.training:__call__] E=2 B=9050 s/sec=1890.19 tok/sec=110953.95 u/sec=24.46 ppl=11.095459 [2023-12-17:11:28:35:INFO:sockeye.training:__call__] E=2 B=9100 s/sec=2358.98 tok/sec=106566.42 u/sec=25.03 ppl=11.130484 [2023-12-17:11:28:37:INFO:sockeye.training:__call__] E=2 B=9150 s/sec=2443.88 tok/sec=103770.99 u/sec=25.54 ppl=11.152482 [2023-12-17:11:28:39:INFO:sockeye.training:__call__] E=2 B=9200 s/sec=2239.02 tok/sec=108776.81 u/sec=24.25 ppl=11.179293 [2023-12-17:11:28:41:INFO:sockeye.training:__call__] E=2 B=9250 s/sec=2020.30 tok/sec=110638.19 u/sec=24.71 ppl=11.187665 [2023-12-17:11:28:43:INFO:sockeye.training:__call__] E=2 B=9300 s/sec=2504.85 tok/sec=101852.82 u/sec=25.92 ppl=11.230388 [2023-12-17:11:28:45:INFO:sockeye.training:__call__] E=2 B=9350 s/sec=2532.65 tok/sec=101138.95 u/sec=26.12 ppl=11.269437 [2023-12-17:11:28:47:INFO:sockeye.training:__call__] E=2 B=9400 s/sec=2109.82 tok/sec=106220.73 u/sec=24.79 ppl=11.285660 [2023-12-17:11:28:49:INFO:sockeye.training:__call__] E=2 B=9450 s/sec=2352.35 tok/sec=103976.26 u/sec=25.48 ppl=11.304235 [2023-12-17:11:28:51:INFO:sockeye.training:__call__] E=2 B=9500 s/sec=2252.21 tok/sec=106954.06 u/sec=25.00 ppl=11.329055 [2023-12-17:11:28:53:INFO:sockeye.training:__call__] E=2 B=9550 s/sec=2314.67 tok/sec=105568.56 u/sec=25.29 ppl=11.350572 [2023-12-17:11:28:54:INFO:sockeye.training:__call__] E=2 B=9600 s/sec=2559.26 tok/sec=104394.79 u/sec=25.51 ppl=11.372803 [2023-12-17:11:28:56:INFO:sockeye.training:__call__] E=2 B=9650 s/sec=2106.31 tok/sec=107381.79 u/sec=25.03 ppl=11.374411 [2023-12-17:11:28:59:INFO:sockeye.training:__call__] E=2 B=9700 s/sec=2178.18 tok/sec=108940.69 u/sec=24.66 ppl=11.400844 [2023-12-17:11:29:01:INFO:sockeye.training:__call__] E=2 B=9750 s/sec=2112.51 tok/sec=111062.08 u/sec=24.27 ppl=11.405187 [2023-12-17:11:29:03:INFO:sockeye.training:__call__] E=2 B=9800 s/sec=2368.40 tok/sec=106006.94 u/sec=25.30 ppl=11.414263 [2023-12-17:11:29:05:INFO:sockeye.training:__call__] E=2 B=9850 s/sec=2060.15 tok/sec=108948.61 u/sec=24.95 ppl=11.420029 [2023-12-17:11:29:07:INFO:sockeye.training:__call__] E=2 B=9900 s/sec=2293.86 tok/sec=109103.12 u/sec=24.47 ppl=11.433916 [2023-12-17:11:29:09:INFO:sockeye.training:__call__] E=2 B=9950 s/sec=2181.97 tok/sec=108826.63 u/sec=24.98 ppl=11.449025 [2023-12-17:11:29:11:INFO:sockeye.training:__call__] E=2 B=10000 s/sec=2090.42 tok/sec=107652.50 u/sec=24.84 ppl=11.464791 [2023-12-17:11:29:13:INFO:sockeye.training:__call__] E=2 B=10050 s/sec=2503.08 tok/sec=102559.52 u/sec=26.07 ppl=11.476800 [2023-12-17:11:29:15:INFO:sockeye.training:__call__] E=2 B=10100 s/sec=2110.38 tok/sec=110670.72 u/sec=24.65 ppl=11.488621 [2023-12-17:11:29:17:INFO:sockeye.training:__call__] E=2 B=10150 s/sec=2312.92 tok/sec=101366.33 u/sec=24.17 ppl=11.509876 [2023-12-17:11:29:19:INFO:sockeye.training:__call__] E=2 B=10200 s/sec=2186.53 tok/sec=109906.97 u/sec=24.19 ppl=11.525590 [2023-12-17:11:29:21:INFO:sockeye.training:__call__] E=2 B=10250 s/sec=2577.72 tok/sec=103599.81 u/sec=25.57 ppl=11.547100 [2023-12-17:11:29:23:INFO:sockeye.training:__call__] E=2 B=10300 s/sec=2261.55 tok/sec=107230.35 u/sec=24.62 ppl=11.566534 [2023-12-17:11:29:25:INFO:sockeye.training:__call__] E=2 B=10350 s/sec=2280.10 tok/sec=106447.49 u/sec=25.18 ppl=11.578456 [2023-12-17:11:29:27:INFO:sockeye.training:__call__] E=2 B=10400 s/sec=2435.08 tok/sec=105651.38 u/sec=25.07 ppl=11.598118 [2023-12-17:11:29:29:INFO:sockeye.training:__call__] E=2 B=10450 s/sec=2315.60 tok/sec=105251.41 u/sec=25.04 ppl=11.613501 [2023-12-17:11:29:31:INFO:sockeye.training:__call__] E=2 B=10500 s/sec=2320.71 tok/sec=103493.92 u/sec=25.72 ppl=11.623477 [2023-12-17:11:29:33:INFO:sockeye.training:__call__] E=2 B=10550 s/sec=2258.98 tok/sec=107075.48 u/sec=25.21 ppl=11.635832 [2023-12-17:11:29:35:INFO:sockeye.training:__call__] E=2 B=10600 s/sec=2341.43 tok/sec=107777.77 u/sec=25.02 ppl=11.655678 [2023-12-17:11:29:37:INFO:sockeye.training:__call__] E=2 B=10650 s/sec=2082.71 tok/sec=106569.52 u/sec=25.52 ppl=11.671093 [2023-12-17:11:29:39:INFO:sockeye.training:__call__] E=2 B=10700 s/sec=2395.14 tok/sec=105516.56 u/sec=25.46 ppl=11.686696 [2023-12-17:11:29:41:INFO:sockeye.training:__call__] E=2 B=10750 s/sec=2087.06 tok/sec=111828.91 u/sec=24.34 ppl=11.701229 [2023-12-17:11:29:43:INFO:sockeye.training:__call__] E=2 B=10800 s/sec=2522.72 tok/sec=104119.53 u/sec=25.60 ppl=11.723277 [2023-12-17:11:29:45:INFO:sockeye.training:__call__] E=2 B=10850 s/sec=2172.57 tok/sec=105022.09 u/sec=25.10 ppl=11.731202 [2023-12-17:11:29:46:INFO:sockeye.training:__call__] E=2 B=10900 s/sec=2158.86 tok/sec=107513.44 u/sec=25.31 ppl=11.742483 [2023-12-17:11:29:48:INFO:sockeye.training:__call__] E=2 B=10950 s/sec=2597.62 tok/sec=104199.48 u/sec=25.57 ppl=11.755462 [2023-12-17:11:29:50:INFO:sockeye.training:__call__] E=2 B=11000 s/sec=2187.46 tok/sec=107104.31 u/sec=25.32 ppl=11.761981 [2023-12-17:11:29:52:INFO:sockeye.training:__call__] E=2 B=11050 s/sec=2229.42 tok/sec=108401.51 u/sec=25.24 ppl=11.772326 [2023-12-17:11:29:54:INFO:sockeye.training:__call__] E=2 B=11100 s/sec=2171.52 tok/sec=102292.43 u/sec=24.06 ppl=11.785421 [2023-12-17:11:29:56:INFO:sockeye.training:__call__] E=2 B=11150 s/sec=2182.69 tok/sec=102864.25 u/sec=24.80 ppl=11.799474 [2023-12-17:11:29:58:INFO:sockeye.training:__call__] E=2 B=11200 s/sec=2542.34 tok/sec=103459.52 u/sec=25.55 ppl=11.811902 [2023-12-17:11:30:00:INFO:sockeye.training:__call__] E=2 B=11250 s/sec=2209.17 tok/sec=106972.04 u/sec=25.10 ppl=11.826524 [2023-12-17:11:30:02:INFO:sockeye.training:__call__] E=2 B=11300 s/sec=2030.25 tok/sec=111976.12 u/sec=24.40 ppl=11.841049 [2023-12-17:11:30:05:INFO:sockeye.training:__call__] E=2 B=11350 s/sec=2227.00 tok/sec=106423.03 u/sec=24.63 ppl=11.850221 [2023-12-17:11:30:07:INFO:sockeye.training:__call__] E=2 B=11400 s/sec=2119.23 tok/sec=105734.43 u/sec=24.48 ppl=11.856158 [2023-12-17:11:30:09:INFO:sockeye.training:__call__] E=2 B=11450 s/sec=2010.59 tok/sec=109376.23 u/sec=24.64 ppl=11.864036 [2023-12-17:11:30:11:INFO:sockeye.training:__call__] E=2 B=11500 s/sec=2191.65 tok/sec=108523.97 u/sec=24.50 ppl=11.876063 [2023-12-17:11:30:13:INFO:sockeye.training:__call__] E=2 B=11550 s/sec=2237.70 tok/sec=106345.41 u/sec=25.20 ppl=11.885707 [2023-12-17:11:30:15:INFO:sockeye.training:__call__] E=2 B=11600 s/sec=2499.67 tok/sec=100906.34 u/sec=25.65 ppl=11.895304 [2023-12-17:11:30:17:INFO:sockeye.training:__call__] E=2 B=11650 s/sec=2032.51 tok/sec=109388.73 u/sec=24.91 ppl=11.901931 [2023-12-17:11:30:19:INFO:sockeye.training:__call__] E=2 B=11700 s/sec=2222.58 tok/sec=108381.97 u/sec=25.03 ppl=11.912242 [2023-12-17:11:30:21:INFO:sockeye.training:__call__] E=2 B=11750 s/sec=2094.99 tok/sec=105985.54 u/sec=25.08 ppl=11.918556 [2023-12-17:11:30:23:INFO:sockeye.training:__call__] E=2 B=11800 s/sec=2104.67 tok/sec=110033.94 u/sec=24.96 ppl=11.929576 [2023-12-17:11:30:25:INFO:sockeye.training:__call__] E=2 B=11850 s/sec=2220.80 tok/sec=106630.60 u/sec=25.19 ppl=11.936036 [2023-12-17:11:30:27:INFO:sockeye.training:__call__] E=2 B=11900 s/sec=2192.94 tok/sec=106281.27 u/sec=25.71 ppl=11.943710 [2023-12-17:11:30:28:INFO:sockeye.training:__call__] E=2 B=11950 s/sec=2365.47 tok/sec=103998.32 u/sec=25.80 ppl=11.952154 [2023-12-17:11:30:31:INFO:sockeye.training:__call__] E=3 B=12000 s/sec=2043.45 tok/sec=101610.43 u/sec=23.74 ppl=11.946895 [2023-12-17:11:30:31:INFO:sockeye.training:_create_checkpoint] Checkpoint [3] Updates=12000 Epoch=3 Samples=1069512 Time-cost=160.446 Updates/sec=24.931 [2023-12-17:11:30:31:INFO:sockeye.training:_create_checkpoint] Checkpoint [3] Train-ppl=11.946895 [2023-12-17:11:30:35:INFO:sockeye.training:_evaluate] Checkpoint [3] Validation-ppl=24.198217 Validation-bleu=0.146511 Validation-chrf=30.119190 Validation-rouge1=0.160281 Validation-rouge2=0.078863 Validation-rougel=0.152774 Validation-length-ratio-mse=1.283743 Validation-ter=95.914156 Validation-avg-sec-per-sent=0.006637 Validation-decode-walltime=3.318676 [2023-12-17:11:30:35:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 1 checkpoints, best so far: 30.600195 [2023-12-17:11:30:35:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [3] Learning-rate=0.000200 [2023-12-17:11:30:35:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:30:35:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:30:36:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00003" [2023-12-17:11:30:38:INFO:sockeye.training:__call__] E=3 B=12050 s/sec=671.04 tok/sec=28916.04 u/sec=7.11 ppl=10.016674 [2023-12-17:11:30:40:INFO:sockeye.training:__call__] E=3 B=12100 s/sec=2178.73 tok/sec=106676.94 u/sec=25.22 ppl=9.851505 [2023-12-17:11:30:42:INFO:sockeye.training:__call__] E=3 B=12150 s/sec=2546.20 tok/sec=102921.63 u/sec=25.42 ppl=9.905411 [2023-12-17:11:30:44:INFO:sockeye.training:__call__] E=3 B=12200 s/sec=1908.69 tok/sec=112826.28 u/sec=24.30 ppl=9.866644 [2023-12-17:11:30:46:INFO:sockeye.training:__call__] E=3 B=12250 s/sec=2087.81 tok/sec=108089.58 u/sec=24.81 ppl=9.814334 [2023-12-17:11:30:48:INFO:sockeye.training:__call__] E=3 B=12300 s/sec=2244.14 tok/sec=107525.81 u/sec=25.09 ppl=9.826744 [2023-12-17:11:30:50:INFO:sockeye.training:__call__] E=3 B=12350 s/sec=2224.54 tok/sec=107265.91 u/sec=25.42 ppl=9.843311 [2023-12-17:11:30:52:INFO:sockeye.training:__call__] E=3 B=12400 s/sec=2307.28 tok/sec=105172.61 u/sec=25.48 ppl=9.872836 [2023-12-17:11:30:54:INFO:sockeye.training:__call__] E=3 B=12450 s/sec=2060.99 tok/sec=108959.27 u/sec=24.72 ppl=9.870092 [2023-12-17:11:30:56:INFO:sockeye.training:__call__] E=3 B=12500 s/sec=2393.78 tok/sec=102199.22 u/sec=25.31 ppl=9.890979 [2023-12-17:11:30:58:INFO:sockeye.training:__call__] E=3 B=12550 s/sec=2133.91 tok/sec=110667.71 u/sec=24.29 ppl=9.915985 [2023-12-17:11:31:00:INFO:sockeye.training:__call__] E=3 B=12600 s/sec=2212.62 tok/sec=104641.94 u/sec=25.99 ppl=9.911805 [2023-12-17:11:31:02:INFO:sockeye.training:__call__] E=3 B=12650 s/sec=2110.78 tok/sec=109792.25 u/sec=24.89 ppl=9.940040 [2023-12-17:11:31:04:INFO:sockeye.training:__call__] E=3 B=12700 s/sec=2235.94 tok/sec=107325.16 u/sec=25.22 ppl=9.959882 [2023-12-17:11:31:06:INFO:sockeye.training:__call__] E=3 B=12750 s/sec=2074.89 tok/sec=108352.51 u/sec=24.70 ppl=9.970109 [2023-12-17:11:31:08:INFO:sockeye.training:__call__] E=3 B=12800 s/sec=2250.92 tok/sec=107172.94 u/sec=25.21 ppl=9.985772 [2023-12-17:11:31:09:INFO:sockeye.training:__call__] E=3 B=12850 s/sec=2236.71 tok/sec=105536.80 u/sec=25.46 ppl=9.982103 [2023-12-17:11:31:12:INFO:sockeye.training:__call__] E=3 B=12900 s/sec=2181.94 tok/sec=109938.80 u/sec=24.35 ppl=10.005415 [2023-12-17:11:31:14:INFO:sockeye.training:__call__] E=3 B=12950 s/sec=2174.26 tok/sec=109760.33 u/sec=24.80 ppl=10.029045 [2023-12-17:11:31:16:INFO:sockeye.training:__call__] E=3 B=13000 s/sec=2437.45 tok/sec=104041.20 u/sec=25.31 ppl=10.071231 [2023-12-17:11:31:18:INFO:sockeye.training:__call__] E=3 B=13050 s/sec=2356.85 tok/sec=105280.59 u/sec=25.44 ppl=10.100793 [2023-12-17:11:31:19:INFO:sockeye.training:__call__] E=3 B=13100 s/sec=2450.26 tok/sec=105785.71 u/sec=25.52 ppl=10.140637 [2023-12-17:11:31:21:INFO:sockeye.training:__call__] E=3 B=13150 s/sec=2143.04 tok/sec=109921.66 u/sec=24.94 ppl=10.164660 [2023-12-17:11:31:24:INFO:sockeye.training:__call__] E=3 B=13200 s/sec=2218.29 tok/sec=113552.07 u/sec=23.82 ppl=10.170193 [2023-12-17:11:31:26:INFO:sockeye.training:__call__] E=3 B=13250 s/sec=2264.51 tok/sec=109640.12 u/sec=24.57 ppl=10.190507 [2023-12-17:11:31:28:INFO:sockeye.training:__call__] E=3 B=13300 s/sec=2358.67 tok/sec=106095.56 u/sec=25.29 ppl=10.215236 [2023-12-17:11:31:29:INFO:sockeye.training:__call__] E=3 B=13350 s/sec=2327.31 tok/sec=100931.62 u/sec=26.40 ppl=10.235486 [2023-12-17:11:31:31:INFO:sockeye.training:__call__] E=3 B=13400 s/sec=2276.14 tok/sec=108174.16 u/sec=24.83 ppl=10.247119 [2023-12-17:11:31:34:INFO:sockeye.training:__call__] E=3 B=13450 s/sec=2003.16 tok/sec=106876.20 u/sec=23.94 ppl=10.256496 [2023-12-17:11:31:36:INFO:sockeye.training:__call__] E=3 B=13500 s/sec=2200.78 tok/sec=108314.83 u/sec=24.61 ppl=10.277170 [2023-12-17:11:31:38:INFO:sockeye.training:__call__] E=3 B=13550 s/sec=2439.04 tok/sec=105450.13 u/sec=25.16 ppl=10.296552 [2023-12-17:11:31:40:INFO:sockeye.training:__call__] E=3 B=13600 s/sec=2250.96 tok/sec=107448.05 u/sec=24.60 ppl=10.318140 [2023-12-17:11:31:42:INFO:sockeye.training:__call__] E=3 B=13650 s/sec=2272.71 tok/sec=107506.46 u/sec=24.75 ppl=10.335599 [2023-12-17:11:31:44:INFO:sockeye.training:__call__] E=3 B=13700 s/sec=2138.68 tok/sec=108317.60 u/sec=24.57 ppl=10.347753 [2023-12-17:11:31:46:INFO:sockeye.training:__call__] E=3 B=13750 s/sec=1978.96 tok/sec=109222.73 u/sec=24.49 ppl=10.354556 [2023-12-17:11:31:48:INFO:sockeye.training:__call__] E=3 B=13800 s/sec=2180.03 tok/sec=107312.39 u/sec=25.14 ppl=10.366620 [2023-12-17:11:31:50:INFO:sockeye.training:__call__] E=3 B=13850 s/sec=2556.85 tok/sec=101737.39 u/sec=26.20 ppl=10.385139 [2023-12-17:11:31:52:INFO:sockeye.training:__call__] E=3 B=13900 s/sec=2069.07 tok/sec=109037.31 u/sec=24.82 ppl=10.390931 [2023-12-17:11:31:54:INFO:sockeye.training:__call__] E=3 B=13950 s/sec=2329.46 tok/sec=105439.35 u/sec=25.41 ppl=10.411995 [2023-12-17:11:31:56:INFO:sockeye.training:__call__] E=3 B=14000 s/sec=2081.09 tok/sec=109552.37 u/sec=24.27 ppl=10.424477 [2023-12-17:11:31:58:INFO:sockeye.training:__call__] E=3 B=14050 s/sec=2264.76 tok/sec=109382.01 u/sec=25.05 ppl=10.435299 [2023-12-17:11:32:00:INFO:sockeye.training:__call__] E=3 B=14100 s/sec=2431.80 tok/sec=104749.55 u/sec=25.29 ppl=10.444945 [2023-12-17:11:32:02:INFO:sockeye.training:__call__] E=3 B=14150 s/sec=2283.66 tok/sec=104959.59 u/sec=25.26 ppl=10.465674 [2023-12-17:11:32:04:INFO:sockeye.training:__call__] E=3 B=14200 s/sec=2393.00 tok/sec=105421.70 u/sec=25.35 ppl=10.477284 [2023-12-17:11:32:06:INFO:sockeye.training:__call__] E=3 B=14250 s/sec=2561.12 tok/sec=101845.96 u/sec=25.99 ppl=10.493326 [2023-12-17:11:32:08:INFO:sockeye.training:__call__] E=3 B=14300 s/sec=2313.67 tok/sec=103807.75 u/sec=25.28 ppl=10.502439 [2023-12-17:11:32:10:INFO:sockeye.training:__call__] E=3 B=14350 s/sec=2222.34 tok/sec=105872.80 u/sec=24.98 ppl=10.511258 [2023-12-17:11:32:12:INFO:sockeye.training:__call__] E=3 B=14400 s/sec=2321.13 tok/sec=106058.53 u/sec=25.06 ppl=10.531826 [2023-12-17:11:32:14:INFO:sockeye.training:__call__] E=3 B=14450 s/sec=2309.55 tok/sec=106851.30 u/sec=24.84 ppl=10.542439 [2023-12-17:11:32:16:INFO:sockeye.training:__call__] E=3 B=14500 s/sec=2196.21 tok/sec=108075.51 u/sec=24.42 ppl=10.554888 [2023-12-17:11:32:18:INFO:sockeye.training:__call__] E=3 B=14550 s/sec=2279.63 tok/sec=106484.72 u/sec=25.22 ppl=10.566321 [2023-12-17:11:32:20:INFO:sockeye.training:__call__] E=3 B=14600 s/sec=2005.44 tok/sec=110112.79 u/sec=24.82 ppl=10.576395 [2023-12-17:11:32:22:INFO:sockeye.training:__call__] E=3 B=14650 s/sec=2224.40 tok/sec=105067.29 u/sec=25.60 ppl=10.584380 [2023-12-17:11:32:23:INFO:sockeye.training:__call__] E=3 B=14700 s/sec=2426.77 tok/sec=103974.20 u/sec=25.32 ppl=10.597297 [2023-12-17:11:32:25:INFO:sockeye.training:__call__] E=3 B=14750 s/sec=2254.17 tok/sec=107516.77 u/sec=25.43 ppl=10.606294 [2023-12-17:11:32:27:INFO:sockeye.training:__call__] E=3 B=14800 s/sec=2248.81 tok/sec=107012.97 u/sec=25.05 ppl=10.617163 [2023-12-17:11:32:30:INFO:sockeye.training:__call__] E=3 B=14850 s/sec=1840.48 tok/sec=112800.46 u/sec=24.06 ppl=10.619899 [2023-12-17:11:32:32:INFO:sockeye.training:__call__] E=3 B=14900 s/sec=2257.81 tok/sec=104153.04 u/sec=24.80 ppl=10.630864 [2023-12-17:11:32:34:INFO:sockeye.training:__call__] E=3 B=14950 s/sec=2068.74 tok/sec=109071.79 u/sec=24.63 ppl=10.642694 [2023-12-17:11:32:36:INFO:sockeye.training:__call__] E=3 B=15000 s/sec=2055.46 tok/sec=108636.51 u/sec=24.90 ppl=10.653150 [2023-12-17:11:32:38:INFO:sockeye.training:__call__] E=3 B=15050 s/sec=2390.22 tok/sec=106466.68 u/sec=25.02 ppl=10.665041 [2023-12-17:11:32:40:INFO:sockeye.training:__call__] E=3 B=15100 s/sec=2171.52 tok/sec=105069.22 u/sec=25.13 ppl=10.678170 [2023-12-17:11:32:42:INFO:sockeye.training:__call__] E=3 B=15150 s/sec=2199.52 tok/sec=107392.91 u/sec=24.46 ppl=10.690890 [2023-12-17:11:32:44:INFO:sockeye.training:__call__] E=3 B=15200 s/sec=2193.95 tok/sec=99490.46 u/sec=24.44 ppl=10.701946 [2023-12-17:11:32:46:INFO:sockeye.training:__call__] E=3 B=15250 s/sec=2205.93 tok/sec=107326.08 u/sec=25.02 ppl=10.705835 [2023-12-17:11:32:48:INFO:sockeye.training:__call__] E=3 B=15300 s/sec=2051.47 tok/sec=107509.66 u/sec=25.04 ppl=10.709276 [2023-12-17:11:32:50:INFO:sockeye.training:__call__] E=3 B=15350 s/sec=2472.70 tok/sec=105044.72 u/sec=25.50 ppl=10.721957 [2023-12-17:11:32:52:INFO:sockeye.training:__call__] E=3 B=15400 s/sec=2237.05 tok/sec=109196.05 u/sec=24.49 ppl=10.736963 [2023-12-17:11:32:54:INFO:sockeye.training:__call__] E=3 B=15450 s/sec=2032.86 tok/sec=109573.99 u/sec=25.06 ppl=10.747530 [2023-12-17:11:32:56:INFO:sockeye.training:__call__] E=3 B=15500 s/sec=2328.54 tok/sec=104286.47 u/sec=25.09 ppl=10.759964 [2023-12-17:11:32:58:INFO:sockeye.training:__call__] E=3 B=15550 s/sec=2141.15 tok/sec=109095.80 u/sec=24.69 ppl=10.768632 [2023-12-17:11:33:00:INFO:sockeye.training:__call__] E=3 B=15600 s/sec=2021.96 tok/sec=108612.50 u/sec=24.88 ppl=10.776463 [2023-12-17:11:33:02:INFO:sockeye.training:__call__] E=3 B=15650 s/sec=2178.40 tok/sec=106566.67 u/sec=24.84 ppl=10.785695 [2023-12-17:11:33:04:INFO:sockeye.training:__call__] E=3 B=15700 s/sec=2110.38 tok/sec=108138.44 u/sec=24.29 ppl=10.795495 [2023-12-17:11:33:06:INFO:sockeye.training:__call__] E=3 B=15750 s/sec=2121.62 tok/sec=102053.13 u/sec=24.02 ppl=10.803107 [2023-12-17:11:33:08:INFO:sockeye.training:__call__] E=3 B=15800 s/sec=2332.58 tok/sec=102808.23 u/sec=24.84 ppl=10.812838 [2023-12-17:11:33:10:INFO:sockeye.training:__call__] E=3 B=15850 s/sec=2311.72 tok/sec=105898.54 u/sec=25.04 ppl=10.821610 [2023-12-17:11:33:12:INFO:sockeye.training:__call__] E=3 B=15900 s/sec=2123.30 tok/sec=107386.35 u/sec=25.28 ppl=10.827376 [2023-12-17:11:33:14:INFO:sockeye.training:__call__] E=3 B=15950 s/sec=2114.06 tok/sec=108718.46 u/sec=24.93 ppl=10.832222 [2023-12-17:11:33:16:INFO:sockeye.training:__call__] E=4 B=16000 s/sec=1899.84 tok/sec=102458.06 u/sec=23.10 ppl=10.824535 [2023-12-17:11:33:16:INFO:sockeye.training:_create_checkpoint] Checkpoint [4] Updates=16000 Epoch=4 Samples=1425712 Time-cost=160.375 Updates/sec=24.942 [2023-12-17:11:33:16:INFO:sockeye.training:_create_checkpoint] Checkpoint [4] Train-ppl=10.824535 [2023-12-17:11:33:19:INFO:sockeye.training:_evaluate] Checkpoint [4] Validation-ppl=25.573629 Validation-bleu=0.157214 Validation-chrf=30.918991 Validation-rouge1=0.172288 Validation-rouge2=0.084025 Validation-rougel=0.165478 Validation-length-ratio-mse=1.274053 Validation-ter=96.904664 Validation-avg-sec-per-sent=0.003402 Validation-decode-walltime=1.700753 [2023-12-17:11:33:19:INFO:sockeye.training:_determine_improvement] Validation-chrf improved to 30.918991 (delta=0.318796). [2023-12-17:11:33:19:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [4] Learning-rate=0.000200 [2023-12-17:11:33:19:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:33:19:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:33:19:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00004" [2023-12-17:11:33:19:INFO:sockeye.training:_update_best_params] '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best' now points to 'params.00004' [2023-12-17:11:33:20:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl" [2023-12-17:11:33:20:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/lr_scheduler_best.pkl' [2023-12-17:11:33:22:INFO:sockeye.training:__call__] E=4 B=16050 s/sec=771.14 tok/sec=35202.42 u/sec=8.53 ppl=8.917675 [2023-12-17:11:33:24:INFO:sockeye.training:__call__] E=4 B=16100 s/sec=2252.91 tok/sec=106902.21 u/sec=24.79 ppl=8.946637 [2023-12-17:11:33:26:INFO:sockeye.training:__call__] E=4 B=16150 s/sec=2149.31 tok/sec=106239.74 u/sec=25.54 ppl=8.860597 [2023-12-17:11:33:28:INFO:sockeye.training:__call__] E=4 B=16200 s/sec=2430.83 tok/sec=99511.69 u/sec=25.07 ppl=8.916239 [2023-12-17:11:33:30:INFO:sockeye.training:__call__] E=4 B=16250 s/sec=2223.41 tok/sec=106818.75 u/sec=24.81 ppl=8.925710 [2023-12-17:11:33:32:INFO:sockeye.training:__call__] E=4 B=16300 s/sec=2122.38 tok/sec=110106.69 u/sec=25.12 ppl=8.944700 [2023-12-17:11:33:34:INFO:sockeye.training:__call__] E=4 B=16350 s/sec=2452.15 tok/sec=103974.31 u/sec=25.84 ppl=8.987162 [2023-12-17:11:33:36:INFO:sockeye.training:__call__] E=4 B=16400 s/sec=1736.69 tok/sec=112767.95 u/sec=24.34 ppl=8.970574 [2023-12-17:11:33:38:INFO:sockeye.training:__call__] E=4 B=16450 s/sec=1954.12 tok/sec=110449.37 u/sec=24.87 ppl=9.003515 [2023-12-17:11:33:40:INFO:sockeye.training:__call__] E=4 B=16500 s/sec=2162.46 tok/sec=108342.25 u/sec=24.48 ppl=9.039726 [2023-12-17:11:33:42:INFO:sockeye.training:__call__] E=4 B=16550 s/sec=2173.87 tok/sec=108140.39 u/sec=24.70 ppl=9.041873 [2023-12-17:11:33:44:INFO:sockeye.training:__call__] E=4 B=16600 s/sec=2093.61 tok/sec=109432.21 u/sec=24.50 ppl=9.060630 [2023-12-17:11:33:46:INFO:sockeye.training:__call__] E=4 B=16650 s/sec=2324.73 tok/sec=105921.04 u/sec=25.01 ppl=9.081475 [2023-12-17:11:33:48:INFO:sockeye.training:__call__] E=4 B=16700 s/sec=2272.62 tok/sec=103570.36 u/sec=25.05 ppl=9.097918 [2023-12-17:11:33:50:INFO:sockeye.training:__call__] E=4 B=16750 s/sec=2388.46 tok/sec=103094.83 u/sec=26.01 ppl=9.112441 [2023-12-17:11:33:52:INFO:sockeye.training:__call__] E=4 B=16800 s/sec=2313.26 tok/sec=106054.21 u/sec=25.28 ppl=9.150787 [2023-12-17:11:33:54:INFO:sockeye.training:__call__] E=4 B=16850 s/sec=2462.46 tok/sec=104587.20 u/sec=25.44 ppl=9.186098 [2023-12-17:11:33:56:INFO:sockeye.training:__call__] E=4 B=16900 s/sec=2522.52 tok/sec=99156.46 u/sec=26.72 ppl=9.227606 [2023-12-17:11:33:58:INFO:sockeye.training:__call__] E=4 B=16950 s/sec=2129.24 tok/sec=108937.35 u/sec=24.02 ppl=9.252101 [2023-12-17:11:34:00:INFO:sockeye.training:__call__] E=4 B=17000 s/sec=2309.28 tok/sec=105892.82 u/sec=25.46 ppl=9.271247 [2023-12-17:11:34:02:INFO:sockeye.training:__call__] E=4 B=17050 s/sec=2225.81 tok/sec=107400.34 u/sec=24.36 ppl=9.290739 [2023-12-17:11:34:04:INFO:sockeye.training:__call__] E=4 B=17100 s/sec=1813.00 tok/sec=118547.46 u/sec=21.83 ppl=9.300319 [2023-12-17:11:34:06:INFO:sockeye.training:__call__] E=4 B=17150 s/sec=2293.33 tok/sec=106254.97 u/sec=25.32 ppl=9.318648 [2023-12-17:11:34:08:INFO:sockeye.training:__call__] E=4 B=17200 s/sec=2198.38 tok/sec=107860.72 u/sec=25.03 ppl=9.341177 [2023-12-17:11:34:10:INFO:sockeye.training:__call__] E=4 B=17250 s/sec=2093.15 tok/sec=109130.41 u/sec=24.87 ppl=9.348260 [2023-12-17:11:34:12:INFO:sockeye.training:__call__] E=4 B=17300 s/sec=2104.07 tok/sec=107488.82 u/sec=24.63 ppl=9.352187 [2023-12-17:11:34:14:INFO:sockeye.training:__call__] E=4 B=17350 s/sec=2103.37 tok/sec=108502.74 u/sec=24.34 ppl=9.371058 [2023-12-17:11:34:16:INFO:sockeye.training:__call__] E=4 B=17400 s/sec=2152.14 tok/sec=107599.28 u/sec=24.86 ppl=9.381219 [2023-12-17:11:34:18:INFO:sockeye.training:__call__] E=4 B=17450 s/sec=2277.84 tok/sec=106942.75 u/sec=24.93 ppl=9.395338 [2023-12-17:11:34:20:INFO:sockeye.training:__call__] E=4 B=17500 s/sec=2250.52 tok/sec=107264.32 u/sec=24.76 ppl=9.411775 [2023-12-17:11:34:22:INFO:sockeye.training:__call__] E=4 B=17550 s/sec=2683.17 tok/sec=100160.30 u/sec=26.28 ppl=9.434743 [2023-12-17:11:34:24:INFO:sockeye.training:__call__] E=4 B=17600 s/sec=2273.10 tok/sec=107523.16 u/sec=25.28 ppl=9.448146 [2023-12-17:11:34:26:INFO:sockeye.training:__call__] E=4 B=17650 s/sec=2487.34 tok/sec=100226.70 u/sec=26.04 ppl=9.465117 [2023-12-17:11:34:28:INFO:sockeye.training:__call__] E=4 B=17700 s/sec=2091.09 tok/sec=109778.24 u/sec=24.66 ppl=9.473233 [2023-12-17:11:34:30:INFO:sockeye.training:__call__] E=4 B=17750 s/sec=2241.21 tok/sec=105772.06 u/sec=25.19 ppl=9.489519 [2023-12-17:11:34:32:INFO:sockeye.training:__call__] E=4 B=17800 s/sec=2160.87 tok/sec=106879.25 u/sec=24.92 ppl=9.503529 [2023-12-17:11:34:34:INFO:sockeye.training:__call__] E=4 B=17850 s/sec=2279.01 tok/sec=107438.06 u/sec=25.03 ppl=9.521187 [2023-12-17:11:34:36:INFO:sockeye.training:__call__] E=4 B=17900 s/sec=2720.37 tok/sec=99492.83 u/sec=26.48 ppl=9.539657 [2023-12-17:11:34:38:INFO:sockeye.training:__call__] E=4 B=17950 s/sec=2338.47 tok/sec=106554.29 u/sec=24.98 ppl=9.554387 [2023-12-17:11:34:40:INFO:sockeye.training:__call__] E=4 B=18000 s/sec=2347.85 tok/sec=105437.40 u/sec=25.43 ppl=9.572145 [2023-12-17:11:34:42:INFO:sockeye.training:__call__] E=4 B=18050 s/sec=2391.69 tok/sec=103701.72 u/sec=25.21 ppl=9.590203 [2023-12-17:11:34:44:INFO:sockeye.training:__call__] E=4 B=18100 s/sec=2168.21 tok/sec=107440.73 u/sec=25.05 ppl=9.610583 [2023-12-17:11:34:46:INFO:sockeye.training:__call__] E=4 B=18150 s/sec=2042.25 tok/sec=109684.38 u/sec=24.88 ppl=9.623043 [2023-12-17:11:34:48:INFO:sockeye.training:__call__] E=4 B=18200 s/sec=2175.97 tok/sec=107826.16 u/sec=24.91 ppl=9.635046 [2023-12-17:11:34:50:INFO:sockeye.training:__call__] E=4 B=18250 s/sec=2500.31 tok/sec=104025.87 u/sec=25.49 ppl=9.652817 [2023-12-17:11:34:52:INFO:sockeye.training:__call__] E=4 B=18300 s/sec=2249.46 tok/sec=103443.29 u/sec=24.75 ppl=9.663840 [2023-12-17:11:34:54:INFO:sockeye.training:__call__] E=4 B=18350 s/sec=2199.50 tok/sec=107928.38 u/sec=25.18 ppl=9.674361 [2023-12-17:11:34:56:INFO:sockeye.training:__call__] E=4 B=18400 s/sec=2177.94 tok/sec=106894.04 u/sec=24.84 ppl=9.687692 [2023-12-17:11:34:58:INFO:sockeye.training:__call__] E=4 B=18450 s/sec=2563.23 tok/sec=101344.24 u/sec=25.71 ppl=9.701103 [2023-12-17:11:35:00:INFO:sockeye.training:__call__] E=4 B=18500 s/sec=2248.70 tok/sec=103510.77 u/sec=24.53 ppl=9.712599 [2023-12-17:11:35:02:INFO:sockeye.training:__call__] E=4 B=18550 s/sec=2169.25 tok/sec=106503.37 u/sec=24.79 ppl=9.722417 [2023-12-17:11:35:04:INFO:sockeye.training:__call__] E=4 B=18600 s/sec=1880.07 tok/sec=112460.90 u/sec=24.38 ppl=9.729283 [2023-12-17:11:35:06:INFO:sockeye.training:__call__] E=4 B=18650 s/sec=2252.22 tok/sec=108106.46 u/sec=25.00 ppl=9.738542 [2023-12-17:11:35:08:INFO:sockeye.training:__call__] E=4 B=18700 s/sec=2211.49 tok/sec=106601.77 u/sec=25.13 ppl=9.749000 [2023-12-17:11:35:10:INFO:sockeye.training:__call__] E=4 B=18750 s/sec=2462.13 tok/sec=105192.81 u/sec=25.56 ppl=9.762095 [2023-12-17:11:35:12:INFO:sockeye.training:__call__] E=4 B=18800 s/sec=2268.63 tok/sec=108607.58 u/sec=24.88 ppl=9.774668 [2023-12-17:11:35:14:INFO:sockeye.training:__call__] E=4 B=18850 s/sec=2135.21 tok/sec=108608.82 u/sec=24.90 ppl=9.781567 [2023-12-17:11:35:16:INFO:sockeye.training:__call__] E=4 B=18900 s/sec=2491.60 tok/sec=102525.83 u/sec=25.70 ppl=9.797903 [2023-12-17:11:35:18:INFO:sockeye.training:__call__] E=4 B=18950 s/sec=1971.41 tok/sec=108458.53 u/sec=24.07 ppl=9.807500 [2023-12-17:11:35:20:INFO:sockeye.training:__call__] E=4 B=19000 s/sec=2266.18 tok/sec=107093.14 u/sec=25.29 ppl=9.816487 [2023-12-17:11:35:22:INFO:sockeye.training:__call__] E=4 B=19050 s/sec=2052.79 tok/sec=110207.43 u/sec=23.94 ppl=9.828563 [2023-12-17:11:35:24:INFO:sockeye.training:__call__] E=4 B=19100 s/sec=2240.77 tok/sec=108261.13 u/sec=25.01 ppl=9.839710 [2023-12-17:11:35:26:INFO:sockeye.training:__call__] E=4 B=19150 s/sec=2068.93 tok/sec=106406.48 u/sec=25.21 ppl=9.843456 [2023-12-17:11:35:28:INFO:sockeye.training:__call__] E=4 B=19200 s/sec=2198.32 tok/sec=107124.06 u/sec=25.07 ppl=9.852151 [2023-12-17:11:35:30:INFO:sockeye.training:__call__] E=4 B=19250 s/sec=2096.44 tok/sec=112203.24 u/sec=23.61 ppl=9.860877 [2023-12-17:11:35:32:INFO:sockeye.training:__call__] E=4 B=19300 s/sec=1930.75 tok/sec=108761.33 u/sec=24.98 ppl=9.868093 [2023-12-17:11:35:34:INFO:sockeye.training:__call__] E=4 B=19350 s/sec=1906.01 tok/sec=111913.08 u/sec=24.36 ppl=9.874656 [2023-12-17:11:35:36:INFO:sockeye.training:__call__] E=4 B=19400 s/sec=2216.72 tok/sec=101624.58 u/sec=25.05 ppl=9.881882 [2023-12-17:11:35:38:INFO:sockeye.training:__call__] E=4 B=19450 s/sec=2440.98 tok/sec=103770.07 u/sec=25.34 ppl=9.892037 [2023-12-17:11:35:40:INFO:sockeye.training:__call__] E=4 B=19500 s/sec=2014.21 tok/sec=106624.04 u/sec=23.75 ppl=9.892601 [2023-12-17:11:35:42:INFO:sockeye.training:__call__] E=4 B=19550 s/sec=1982.70 tok/sec=110847.39 u/sec=23.92 ppl=9.902892 [2023-12-17:11:35:44:INFO:sockeye.training:__call__] E=4 B=19600 s/sec=2161.43 tok/sec=109054.80 u/sec=24.38 ppl=9.910184 [2023-12-17:11:35:46:INFO:sockeye.training:__call__] E=4 B=19650 s/sec=2406.30 tok/sec=104727.60 u/sec=25.66 ppl=9.921298 [2023-12-17:11:35:48:INFO:sockeye.training:__call__] E=4 B=19700 s/sec=2216.72 tok/sec=109331.50 u/sec=24.87 ppl=9.933149 [2023-12-17:11:35:50:INFO:sockeye.training:__call__] E=4 B=19750 s/sec=2465.94 tok/sec=102569.89 u/sec=25.82 ppl=9.945330 [2023-12-17:11:35:52:INFO:sockeye.training:__call__] E=4 B=19800 s/sec=2284.72 tok/sec=106319.92 u/sec=25.14 ppl=9.954517 [2023-12-17:11:35:54:INFO:sockeye.training:__call__] E=4 B=19850 s/sec=2158.32 tok/sec=105474.96 u/sec=25.26 ppl=9.964184 [2023-12-17:11:35:56:INFO:sockeye.training:__call__] E=4 B=19900 s/sec=2324.00 tok/sec=107286.01 u/sec=24.87 ppl=9.972488 [2023-12-17:11:35:58:INFO:sockeye.training:__call__] E=4 B=19950 s/sec=2215.14 tok/sec=107321.80 u/sec=25.08 ppl=9.981105 [2023-12-17:11:36:00:INFO:sockeye.training:__call__] E=5 B=20000 s/sec=1926.05 tok/sec=102612.58 u/sec=23.42 ppl=9.971819 [2023-12-17:11:36:00:INFO:sockeye.training:_create_checkpoint] Checkpoint [5] Updates=20000 Epoch=5 Samples=1782008 Time-cost=160.425 Updates/sec=24.934 [2023-12-17:11:36:00:INFO:sockeye.training:_create_checkpoint] Checkpoint [5] Train-ppl=9.971819 [2023-12-17:11:36:03:INFO:sockeye.training:_evaluate] Checkpoint [5] Validation-ppl=26.137648 Validation-bleu=0.175208 Validation-chrf=31.983236 Validation-rouge1=0.168988 Validation-rouge2=0.084620 Validation-rougel=0.163078 Validation-length-ratio-mse=1.204812 Validation-ter=92.695006 Validation-avg-sec-per-sent=0.003114 Validation-decode-walltime=1.557038 [2023-12-17:11:36:03:INFO:sockeye.training:_determine_improvement] Validation-chrf improved to 31.983236 (delta=1.064246). [2023-12-17:11:36:03:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [5] Learning-rate=0.000200 [2023-12-17:11:36:03:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:36:03:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:36:04:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00005" [2023-12-17:11:36:04:INFO:sockeye.training:_update_best_params] '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best' now points to 'params.00005' [2023-12-17:11:36:04:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl" [2023-12-17:11:36:04:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/lr_scheduler_best.pkl' [2023-12-17:11:36:06:INFO:sockeye.training:__call__] E=5 B=20050 s/sec=692.68 tok/sec=37995.43 u/sec=8.51 ppl=8.312732 [2023-12-17:11:36:08:INFO:sockeye.training:__call__] E=5 B=20100 s/sec=2389.29 tok/sec=103928.09 u/sec=25.01 ppl=8.364562 [2023-12-17:11:36:10:INFO:sockeye.training:__call__] E=5 B=20150 s/sec=2356.27 tok/sec=104608.90 u/sec=25.13 ppl=8.402394 [2023-12-17:11:36:12:INFO:sockeye.training:__call__] E=5 B=20200 s/sec=1927.64 tok/sec=107883.65 u/sec=25.05 ppl=8.390674 [2023-12-17:11:36:14:INFO:sockeye.training:__call__] E=5 B=20250 s/sec=2147.02 tok/sec=107923.58 u/sec=24.85 ppl=8.405565 [2023-12-17:11:36:16:INFO:sockeye.training:__call__] E=5 B=20300 s/sec=2277.79 tok/sec=103714.62 u/sec=24.80 ppl=8.451565 [2023-12-17:11:36:18:INFO:sockeye.training:__call__] E=5 B=20350 s/sec=2225.03 tok/sec=106379.76 u/sec=25.33 ppl=8.474474 [2023-12-17:11:36:20:INFO:sockeye.training:__call__] E=5 B=20400 s/sec=2289.14 tok/sec=106111.06 u/sec=25.59 ppl=8.485134 [2023-12-17:11:36:22:INFO:sockeye.training:__call__] E=5 B=20450 s/sec=1969.36 tok/sec=108533.64 u/sec=24.42 ppl=8.472321 [2023-12-17:11:36:24:INFO:sockeye.training:__call__] E=5 B=20500 s/sec=2156.61 tok/sec=106786.74 u/sec=25.29 ppl=8.493707 [2023-12-17:11:36:26:INFO:sockeye.training:__call__] E=5 B=20550 s/sec=2184.64 tok/sec=107618.94 u/sec=25.33 ppl=8.546997 [2023-12-17:11:36:28:INFO:sockeye.training:__call__] E=5 B=20600 s/sec=2259.68 tok/sec=107860.82 u/sec=24.82 ppl=8.568103 [2023-12-17:11:36:30:INFO:sockeye.training:__call__] E=5 B=20650 s/sec=1972.55 tok/sec=110805.94 u/sec=24.36 ppl=8.580967 [2023-12-17:11:36:32:INFO:sockeye.training:__call__] E=5 B=20700 s/sec=2168.08 tok/sec=108522.65 u/sec=24.68 ppl=8.606113 [2023-12-17:11:36:34:INFO:sockeye.training:__call__] E=5 B=20750 s/sec=2365.79 tok/sec=106335.59 u/sec=25.19 ppl=8.632571 [2023-12-17:11:36:36:INFO:sockeye.training:__call__] E=5 B=20800 s/sec=2321.75 tok/sec=103943.03 u/sec=25.37 ppl=8.641954 [2023-12-17:11:36:38:INFO:sockeye.training:__call__] E=5 B=20850 s/sec=2267.36 tok/sec=107193.01 u/sec=24.65 ppl=8.656347 [2023-12-17:11:36:40:INFO:sockeye.training:__call__] E=5 B=20900 s/sec=2186.51 tok/sec=105804.19 u/sec=23.77 ppl=8.676350 [2023-12-17:11:36:42:INFO:sockeye.training:__call__] E=5 B=20950 s/sec=2174.18 tok/sec=105698.43 u/sec=24.89 ppl=8.686694 [2023-12-17:11:36:44:INFO:sockeye.training:__call__] E=5 B=21000 s/sec=2192.09 tok/sec=108290.87 u/sec=24.73 ppl=8.705998 [2023-12-17:11:36:46:INFO:sockeye.training:__call__] E=5 B=21050 s/sec=2418.15 tok/sec=102075.97 u/sec=25.49 ppl=8.728302 [2023-12-17:11:36:48:INFO:sockeye.training:__call__] E=5 B=21100 s/sec=2008.18 tok/sec=102323.24 u/sec=24.51 ppl=8.736501 [2023-12-17:11:36:50:INFO:sockeye.training:__call__] E=5 B=21150 s/sec=2171.88 tok/sec=102631.95 u/sec=23.86 ppl=8.743320 [2023-12-17:11:36:52:INFO:sockeye.training:__call__] E=5 B=21200 s/sec=2316.54 tok/sec=105027.32 u/sec=25.76 ppl=8.760357 [2023-12-17:11:36:54:INFO:sockeye.training:__call__] E=5 B=21250 s/sec=2493.44 tok/sec=100978.01 u/sec=26.19 ppl=8.775830 [2023-12-17:11:36:56:INFO:sockeye.training:__call__] E=5 B=21300 s/sec=2258.62 tok/sec=107548.72 u/sec=25.03 ppl=8.783487 [2023-12-17:11:36:58:INFO:sockeye.training:__call__] E=5 B=21350 s/sec=2410.73 tok/sec=103730.77 u/sec=25.58 ppl=8.794880 [2023-12-17:11:37:00:INFO:sockeye.training:__call__] E=5 B=21400 s/sec=2240.77 tok/sec=109675.28 u/sec=24.70 ppl=8.811026 [2023-12-17:11:37:02:INFO:sockeye.training:__call__] E=5 B=21450 s/sec=2240.18 tok/sec=107307.20 u/sec=24.69 ppl=8.824652 [2023-12-17:11:37:04:INFO:sockeye.training:__call__] E=5 B=21500 s/sec=2619.40 tok/sec=100545.29 u/sec=25.82 ppl=8.846253 [2023-12-17:11:37:06:INFO:sockeye.training:__call__] E=5 B=21550 s/sec=2204.88 tok/sec=100635.32 u/sec=24.18 ppl=8.862228 [2023-12-17:11:37:08:INFO:sockeye.training:__call__] E=5 B=21600 s/sec=2504.01 tok/sec=95265.14 u/sec=25.16 ppl=8.883150 [2023-12-17:11:37:10:INFO:sockeye.training:__call__] E=5 B=21650 s/sec=2078.43 tok/sec=111029.23 u/sec=24.79 ppl=8.891582 [2023-12-17:11:37:12:INFO:sockeye.training:__call__] E=5 B=21700 s/sec=2217.76 tok/sec=108567.47 u/sec=24.66 ppl=8.906115 [2023-12-17:11:37:14:INFO:sockeye.training:__call__] E=5 B=21750 s/sec=2256.01 tok/sec=113718.05 u/sec=23.31 ppl=8.912837 [2023-12-17:11:37:16:INFO:sockeye.training:__call__] E=5 B=21800 s/sec=2355.14 tok/sec=105050.71 u/sec=25.29 ppl=8.924348 [2023-12-17:11:37:18:INFO:sockeye.training:__call__] E=5 B=21850 s/sec=2433.65 tok/sec=103748.86 u/sec=25.39 ppl=8.938218 [2023-12-17:11:37:20:INFO:sockeye.training:__call__] E=5 B=21900 s/sec=2092.80 tok/sec=104592.11 u/sec=24.87 ppl=8.952455 [2023-12-17:11:37:22:INFO:sockeye.training:__call__] E=5 B=21950 s/sec=2040.32 tok/sec=111068.69 u/sec=24.43 ppl=8.958692 [2023-12-17:11:37:24:INFO:sockeye.training:__call__] E=5 B=22000 s/sec=2166.82 tok/sec=105997.70 u/sec=25.08 ppl=8.971909 [2023-12-17:11:37:27:INFO:sockeye.training:__call__] E=5 B=22050 s/sec=1938.78 tok/sec=108789.97 u/sec=24.33 ppl=8.977182 [2023-12-17:11:37:29:INFO:sockeye.training:__call__] E=5 B=22100 s/sec=2038.51 tok/sec=110103.30 u/sec=24.93 ppl=8.984316 [2023-12-17:11:37:31:INFO:sockeye.training:__call__] E=5 B=22150 s/sec=2120.40 tok/sec=107756.63 u/sec=25.24 ppl=8.994552 [2023-12-17:11:37:33:INFO:sockeye.training:__call__] E=5 B=22200 s/sec=2262.75 tok/sec=105819.62 u/sec=25.07 ppl=9.007369 [2023-12-17:11:37:34:INFO:sockeye.training:__call__] E=5 B=22250 s/sec=2289.04 tok/sec=106270.12 u/sec=25.37 ppl=9.019491 [2023-12-17:11:37:36:INFO:sockeye.training:__call__] E=5 B=22300 s/sec=2230.77 tok/sec=106328.01 u/sec=25.44 ppl=9.028051 [2023-12-17:11:37:38:INFO:sockeye.training:__call__] E=5 B=22350 s/sec=2217.77 tok/sec=105634.69 u/sec=25.57 ppl=9.040839 [2023-12-17:11:37:40:INFO:sockeye.training:__call__] E=5 B=22400 s/sec=2482.71 tok/sec=103761.75 u/sec=25.40 ppl=9.054281 [2023-12-17:11:37:42:INFO:sockeye.training:__call__] E=5 B=22450 s/sec=2480.24 tok/sec=100650.17 u/sec=25.58 ppl=9.069582 [2023-12-17:11:37:44:INFO:sockeye.training:__call__] E=5 B=22500 s/sec=2144.12 tok/sec=106602.30 u/sec=24.82 ppl=9.082851 [2023-12-17:11:37:46:INFO:sockeye.training:__call__] E=5 B=22550 s/sec=2259.99 tok/sec=106611.46 u/sec=24.74 ppl=9.093250 [2023-12-17:11:37:48:INFO:sockeye.training:__call__] E=5 B=22600 s/sec=2438.50 tok/sec=105284.34 u/sec=25.53 ppl=9.108961 [2023-12-17:11:37:50:INFO:sockeye.training:__call__] E=5 B=22650 s/sec=2590.35 tok/sec=103219.86 u/sec=25.66 ppl=9.121234 [2023-12-17:11:37:52:INFO:sockeye.training:__call__] E=5 B=22700 s/sec=2114.71 tok/sec=108398.28 u/sec=24.48 ppl=9.127049 [2023-12-17:11:37:54:INFO:sockeye.training:__call__] E=5 B=22750 s/sec=2183.61 tok/sec=107946.52 u/sec=24.72 ppl=9.138342 [2023-12-17:11:37:56:INFO:sockeye.training:__call__] E=5 B=22800 s/sec=2079.07 tok/sec=110182.81 u/sec=24.89 ppl=9.143373 [2023-12-17:11:37:58:INFO:sockeye.training:__call__] E=5 B=22850 s/sec=2017.33 tok/sec=108729.90 u/sec=24.72 ppl=9.153357 [2023-12-17:11:38:00:INFO:sockeye.training:__call__] E=5 B=22900 s/sec=2001.94 tok/sec=105405.29 u/sec=23.70 ppl=9.162917 [2023-12-17:11:38:02:INFO:sockeye.training:__call__] E=5 B=22950 s/sec=2322.05 tok/sec=107736.83 u/sec=25.28 ppl=9.171681 [2023-12-17:11:38:04:INFO:sockeye.training:__call__] E=5 B=23000 s/sec=2464.79 tok/sec=105087.44 u/sec=25.76 ppl=9.187783 [2023-12-17:11:38:06:INFO:sockeye.training:__call__] E=5 B=23050 s/sec=2452.78 tok/sec=104824.24 u/sec=25.21 ppl=9.197349 [2023-12-17:11:38:08:INFO:sockeye.training:__call__] E=5 B=23100 s/sec=2422.48 tok/sec=105652.55 u/sec=25.23 ppl=9.208992 [2023-12-17:11:38:10:INFO:sockeye.training:__call__] E=5 B=23150 s/sec=2074.42 tok/sec=112991.72 u/sec=24.32 ppl=9.214517 [2023-12-17:11:38:12:INFO:sockeye.training:__call__] E=5 B=23200 s/sec=2110.53 tok/sec=108040.41 u/sec=24.47 ppl=9.224298 [2023-12-17:11:38:14:INFO:sockeye.training:__call__] E=5 B=23250 s/sec=2349.66 tok/sec=106873.06 u/sec=25.23 ppl=9.234555 [2023-12-17:11:38:16:INFO:sockeye.training:__call__] E=5 B=23300 s/sec=2106.73 tok/sec=107979.86 u/sec=25.03 ppl=9.240308 [2023-12-17:11:38:19:INFO:sockeye.training:__call__] E=5 B=23350 s/sec=2064.57 tok/sec=108985.02 u/sec=23.76 ppl=9.252122 [2023-12-17:11:38:21:INFO:sockeye.training:__call__] E=5 B=23400 s/sec=2102.04 tok/sec=103484.88 u/sec=24.06 ppl=9.259168 [2023-12-17:11:38:23:INFO:sockeye.training:__call__] E=5 B=23450 s/sec=2199.16 tok/sec=109778.22 u/sec=24.41 ppl=9.268920 [2023-12-17:11:38:25:INFO:sockeye.training:__call__] E=5 B=23500 s/sec=2003.15 tok/sec=105530.14 u/sec=24.26 ppl=9.274381 [2023-12-17:11:38:27:INFO:sockeye.training:__call__] E=5 B=23550 s/sec=2311.77 tok/sec=106672.17 u/sec=25.22 ppl=9.283502 [2023-12-17:11:38:29:INFO:sockeye.training:__call__] E=5 B=23600 s/sec=2318.70 tok/sec=105915.13 u/sec=25.03 ppl=9.293539 [2023-12-17:11:38:31:INFO:sockeye.training:__call__] E=5 B=23650 s/sec=2149.05 tok/sec=109112.86 u/sec=24.24 ppl=9.302120 [2023-12-17:11:38:33:INFO:sockeye.training:__call__] E=5 B=23700 s/sec=2275.59 tok/sec=107288.30 u/sec=25.26 ppl=9.310158 [2023-12-17:11:38:35:INFO:sockeye.training:__call__] E=5 B=23750 s/sec=2202.79 tok/sec=106553.01 u/sec=25.59 ppl=9.317228 [2023-12-17:11:38:37:INFO:sockeye.training:__call__] E=5 B=23800 s/sec=2222.14 tok/sec=101661.02 u/sec=24.89 ppl=9.321441 [2023-12-17:11:38:39:INFO:sockeye.training:__call__] E=5 B=23850 s/sec=2154.59 tok/sec=108062.35 u/sec=24.18 ppl=9.329082 [2023-12-17:11:38:41:INFO:sockeye.training:__call__] E=5 B=23900 s/sec=1968.41 tok/sec=112029.79 u/sec=24.65 ppl=9.333308 [2023-12-17:11:38:43:INFO:sockeye.training:__call__] E=5 B=23950 s/sec=2191.88 tok/sec=105427.15 u/sec=24.20 ppl=9.345251 [2023-12-17:11:38:45:INFO:sockeye.training:__call__] E=6 B=24000 s/sec=1858.21 tok/sec=98942.76 u/sec=22.33 ppl=9.335833 [2023-12-17:11:38:45:INFO:sockeye.training:_create_checkpoint] Checkpoint [6] Updates=24000 Epoch=6 Samples=2138832 Time-cost=160.980 Updates/sec=24.848 [2023-12-17:11:38:45:INFO:sockeye.training:_create_checkpoint] Checkpoint [6] Train-ppl=9.335833 [2023-12-17:11:38:48:INFO:sockeye.training:_evaluate] Checkpoint [6] Validation-ppl=27.926656 Validation-bleu=0.139581 Validation-chrf=29.959064 Validation-rouge1=0.159920 Validation-rouge2=0.072936 Validation-rougel=0.152293 Validation-length-ratio-mse=1.290122 Validation-ter=97.730087 Validation-avg-sec-per-sent=0.003217 Validation-decode-walltime=1.608505 [2023-12-17:11:38:48:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 1 checkpoints, best so far: 31.983236 [2023-12-17:11:38:48:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [6] Learning-rate=0.000200 [2023-12-17:11:38:48:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:38:48:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:38:48:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00006" [2023-12-17:11:38:51:INFO:sockeye.training:__call__] E=6 B=24050 s/sec=784.72 tok/sec=40270.44 u/sec=9.29 ppl=7.673105 [2023-12-17:11:38:53:INFO:sockeye.training:__call__] E=6 B=24100 s/sec=2055.91 tok/sec=109130.34 u/sec=24.29 ppl=7.784262 [2023-12-17:11:38:55:INFO:sockeye.training:__call__] E=6 B=24150 s/sec=2162.08 tok/sec=107251.93 u/sec=25.35 ppl=7.812009 [2023-12-17:11:38:57:INFO:sockeye.training:__call__] E=6 B=24200 s/sec=2320.61 tok/sec=105178.88 u/sec=25.67 ppl=7.860158 [2023-12-17:11:38:58:INFO:sockeye.training:__call__] E=6 B=24250 s/sec=2254.86 tok/sec=106710.96 u/sec=25.30 ppl=7.895305 [2023-12-17:11:39:01:INFO:sockeye.training:__call__] E=6 B=24300 s/sec=2183.41 tok/sec=110175.99 u/sec=24.54 ppl=7.891324 [2023-12-17:11:39:03:INFO:sockeye.training:__call__] E=6 B=24350 s/sec=2056.01 tok/sec=102717.67 u/sec=23.49 ppl=7.916561 [2023-12-17:11:39:05:INFO:sockeye.training:__call__] E=6 B=24400 s/sec=1938.58 tok/sec=110644.39 u/sec=23.94 ppl=7.929631 [2023-12-17:11:39:07:INFO:sockeye.training:__call__] E=6 B=24450 s/sec=2046.00 tok/sec=104064.57 u/sec=23.46 ppl=7.943886 [2023-12-17:11:39:09:INFO:sockeye.training:__call__] E=6 B=24500 s/sec=2285.32 tok/sec=107180.48 u/sec=25.19 ppl=7.967228 [2023-12-17:11:39:11:INFO:sockeye.training:__call__] E=6 B=24550 s/sec=2044.22 tok/sec=111777.23 u/sec=24.81 ppl=7.974400 [2023-12-17:11:39:13:INFO:sockeye.training:__call__] E=6 B=24600 s/sec=2315.87 tok/sec=114254.58 u/sec=23.69 ppl=8.001292 [2023-12-17:11:39:15:INFO:sockeye.training:__call__] E=6 B=24650 s/sec=2062.84 tok/sec=108026.62 u/sec=24.70 ppl=7.997088 [2023-12-17:11:39:17:INFO:sockeye.training:__call__] E=6 B=24700 s/sec=2304.67 tok/sec=104908.91 u/sec=25.23 ppl=8.017593 [2023-12-17:11:39:19:INFO:sockeye.training:__call__] E=6 B=24750 s/sec=2129.88 tok/sec=105603.35 u/sec=25.31 ppl=8.019784 [2023-12-17:11:39:21:INFO:sockeye.training:__call__] E=6 B=24800 s/sec=2396.46 tok/sec=105542.64 u/sec=25.65 ppl=8.038520 [2023-12-17:11:39:23:INFO:sockeye.training:__call__] E=6 B=24850 s/sec=2309.01 tok/sec=103656.27 u/sec=25.96 ppl=8.060672 [2023-12-17:11:39:25:INFO:sockeye.training:__call__] E=6 B=24900 s/sec=2440.94 tok/sec=105378.12 u/sec=25.09 ppl=8.090107 [2023-12-17:11:39:27:INFO:sockeye.training:__call__] E=6 B=24950 s/sec=2269.86 tok/sec=110438.39 u/sec=24.17 ppl=8.106067 [2023-12-17:11:39:29:INFO:sockeye.training:__call__] E=6 B=25000 s/sec=2211.45 tok/sec=110360.44 u/sec=24.55 ppl=8.127705 [2023-12-17:11:39:31:INFO:sockeye.training:__call__] E=6 B=25050 s/sec=2204.47 tok/sec=108599.20 u/sec=25.01 ppl=8.144946 [2023-12-17:11:39:33:INFO:sockeye.training:__call__] E=6 B=25100 s/sec=2267.91 tok/sec=105185.70 u/sec=25.40 ppl=8.164599 [2023-12-17:11:39:35:INFO:sockeye.training:__call__] E=6 B=25150 s/sec=2554.67 tok/sec=104457.57 u/sec=25.71 ppl=8.187195 [2023-12-17:11:39:37:INFO:sockeye.training:__call__] E=6 B=25200 s/sec=2247.17 tok/sec=106368.84 u/sec=24.86 ppl=8.208164 [2023-12-17:11:39:39:INFO:sockeye.training:__call__] E=6 B=25250 s/sec=2089.13 tok/sec=110142.57 u/sec=24.54 ppl=8.223114 [2023-12-17:11:39:41:INFO:sockeye.training:__call__] E=6 B=25300 s/sec=2208.82 tok/sec=106435.76 u/sec=24.78 ppl=8.234559 [2023-12-17:11:39:43:INFO:sockeye.training:__call__] E=6 B=25350 s/sec=2097.08 tok/sec=108119.50 u/sec=25.01 ppl=8.246356 [2023-12-17:11:39:45:INFO:sockeye.training:__call__] E=6 B=25400 s/sec=2412.70 tok/sec=104248.38 u/sec=25.52 ppl=8.260521 [2023-12-17:11:39:47:INFO:sockeye.training:__call__] E=6 B=25450 s/sec=1968.78 tok/sec=112122.98 u/sec=24.37 ppl=8.264884 [2023-12-17:11:39:49:INFO:sockeye.training:__call__] E=6 B=25500 s/sec=2229.03 tok/sec=107470.86 u/sec=24.88 ppl=8.280757 [2023-12-17:11:39:51:INFO:sockeye.training:__call__] E=6 B=25550 s/sec=2561.16 tok/sec=102577.88 u/sec=25.69 ppl=8.300114 [2023-12-17:11:39:53:INFO:sockeye.training:__call__] E=6 B=25600 s/sec=2322.85 tok/sec=102658.79 u/sec=25.29 ppl=8.313323 [2023-12-17:11:39:55:INFO:sockeye.training:__call__] E=6 B=25650 s/sec=1994.53 tok/sec=103925.71 u/sec=23.43 ppl=8.317091 [2023-12-17:11:39:57:INFO:sockeye.training:__call__] E=6 B=25700 s/sec=2392.67 tok/sec=103924.15 u/sec=25.48 ppl=8.331420 [2023-12-17:11:39:59:INFO:sockeye.training:__call__] E=6 B=25750 s/sec=2286.64 tok/sec=106488.58 u/sec=25.29 ppl=8.348136 [2023-12-17:11:40:01:INFO:sockeye.training:__call__] E=6 B=25800 s/sec=1955.66 tok/sec=112498.29 u/sec=24.01 ppl=8.353168 [2023-12-17:11:40:03:INFO:sockeye.training:__call__] E=6 B=25850 s/sec=2242.29 tok/sec=107341.78 u/sec=25.03 ppl=8.368899 [2023-12-17:11:40:05:INFO:sockeye.training:__call__] E=6 B=25900 s/sec=2114.32 tok/sec=106219.74 u/sec=24.98 ppl=8.380126 [2023-12-17:11:40:07:INFO:sockeye.training:__call__] E=6 B=25950 s/sec=1855.60 tok/sec=110948.48 u/sec=24.21 ppl=8.388803 [2023-12-17:11:40:09:INFO:sockeye.training:__call__] E=6 B=26000 s/sec=2548.18 tok/sec=101662.26 u/sec=25.85 ppl=8.405251 [2023-12-17:11:40:11:INFO:sockeye.training:__call__] E=6 B=26050 s/sec=2094.36 tok/sec=104583.16 u/sec=24.74 ppl=8.419381 [2023-12-17:11:40:13:INFO:sockeye.training:__call__] E=6 B=26100 s/sec=2285.86 tok/sec=100453.26 u/sec=24.30 ppl=8.431631 [2023-12-17:11:40:15:INFO:sockeye.training:__call__] E=6 B=26150 s/sec=2201.22 tok/sec=105106.36 u/sec=23.97 ppl=8.447868 [2023-12-17:11:40:17:INFO:sockeye.training:__call__] E=6 B=26200 s/sec=2046.49 tok/sec=109042.15 u/sec=24.41 ppl=8.457952 [2023-12-17:11:40:19:INFO:sockeye.training:__call__] E=6 B=26250 s/sec=2405.14 tok/sec=106004.19 u/sec=25.26 ppl=8.473095 [2023-12-17:11:40:21:INFO:sockeye.training:__call__] E=6 B=26300 s/sec=2250.39 tok/sec=105483.23 u/sec=24.76 ppl=8.483119 [2023-12-17:11:40:23:INFO:sockeye.training:__call__] E=6 B=26350 s/sec=2122.39 tok/sec=109572.14 u/sec=24.75 ppl=8.492951 [2023-12-17:11:40:25:INFO:sockeye.training:__call__] E=6 B=26400 s/sec=2216.60 tok/sec=109426.66 u/sec=24.92 ppl=8.503447 [2023-12-17:11:40:27:INFO:sockeye.training:__call__] E=6 B=26450 s/sec=2032.77 tok/sec=110102.03 u/sec=24.72 ppl=8.515199 [2023-12-17:11:40:29:INFO:sockeye.training:__call__] E=6 B=26500 s/sec=2255.74 tok/sec=106362.68 u/sec=24.91 ppl=8.524718 [2023-12-17:11:40:31:INFO:sockeye.training:__call__] E=6 B=26550 s/sec=2101.10 tok/sec=104566.19 u/sec=25.45 ppl=8.534704 [2023-12-17:11:40:33:INFO:sockeye.training:__call__] E=6 B=26600 s/sec=2212.05 tok/sec=108311.11 u/sec=24.87 ppl=8.547652 [2023-12-17:11:40:35:INFO:sockeye.training:__call__] E=6 B=26650 s/sec=2127.60 tok/sec=105381.86 u/sec=24.95 ppl=8.554857 [2023-12-17:11:40:37:INFO:sockeye.training:__call__] E=6 B=26700 s/sec=2290.11 tok/sec=104976.28 u/sec=25.60 ppl=8.569227 [2023-12-17:11:40:39:INFO:sockeye.training:__call__] E=6 B=26750 s/sec=2536.01 tok/sec=98588.22 u/sec=25.32 ppl=8.582705 [2023-12-17:11:40:41:INFO:sockeye.training:__call__] E=6 B=26800 s/sec=2186.35 tok/sec=103746.00 u/sec=24.02 ppl=8.594748 [2023-12-17:11:40:43:INFO:sockeye.training:__call__] E=6 B=26850 s/sec=2444.83 tok/sec=102190.62 u/sec=25.64 ppl=8.608649 [2023-12-17:11:40:45:INFO:sockeye.training:__call__] E=6 B=26900 s/sec=2216.67 tok/sec=106976.54 u/sec=25.01 ppl=8.620716 [2023-12-17:11:40:47:INFO:sockeye.training:__call__] E=6 B=26950 s/sec=2468.92 tok/sec=105752.79 u/sec=25.42 ppl=8.632968 [2023-12-17:11:40:49:INFO:sockeye.training:__call__] E=6 B=27000 s/sec=2131.44 tok/sec=108679.64 u/sec=24.76 ppl=8.645294 [2023-12-17:11:40:51:INFO:sockeye.training:__call__] E=6 B=27050 s/sec=2196.28 tok/sec=106743.20 u/sec=25.19 ppl=8.656513 [2023-12-17:11:40:53:INFO:sockeye.training:__call__] E=6 B=27100 s/sec=2192.84 tok/sec=106196.84 u/sec=25.33 ppl=8.665638 [2023-12-17:11:40:55:INFO:sockeye.training:__call__] E=6 B=27150 s/sec=2181.22 tok/sec=108519.48 u/sec=24.88 ppl=8.675521 [2023-12-17:11:40:57:INFO:sockeye.training:__call__] E=6 B=27200 s/sec=2328.91 tok/sec=102714.14 u/sec=25.23 ppl=8.685204 [2023-12-17:11:40:59:INFO:sockeye.training:__call__] E=6 B=27250 s/sec=2370.29 tok/sec=106433.64 u/sec=25.15 ppl=8.696401 [2023-12-17:11:41:01:INFO:sockeye.training:__call__] E=6 B=27300 s/sec=2159.96 tok/sec=108111.20 u/sec=25.28 ppl=8.706694 [2023-12-17:11:41:03:INFO:sockeye.training:__call__] E=6 B=27350 s/sec=2176.31 tok/sec=108591.62 u/sec=25.00 ppl=8.715025 [2023-12-17:11:41:05:INFO:sockeye.training:__call__] E=6 B=27400 s/sec=2274.45 tok/sec=105229.00 u/sec=24.85 ppl=8.723289 [2023-12-17:11:41:07:INFO:sockeye.training:__call__] E=6 B=27450 s/sec=1770.27 tok/sec=114264.11 u/sec=24.37 ppl=8.729853 [2023-12-17:11:41:09:INFO:sockeye.training:__call__] E=6 B=27500 s/sec=2316.60 tok/sec=105193.39 u/sec=25.49 ppl=8.741081 [2023-12-17:11:41:11:INFO:sockeye.training:__call__] E=6 B=27550 s/sec=2561.73 tok/sec=101096.36 u/sec=26.16 ppl=8.752131 [2023-12-17:11:41:13:INFO:sockeye.training:__call__] E=6 B=27600 s/sec=2487.98 tok/sec=105586.46 u/sec=25.08 ppl=8.764073 [2023-12-17:11:41:15:INFO:sockeye.training:__call__] E=6 B=27650 s/sec=2320.50 tok/sec=105715.98 u/sec=24.88 ppl=8.770549 [2023-12-17:11:41:17:INFO:sockeye.training:__call__] E=6 B=27700 s/sec=2048.75 tok/sec=110899.13 u/sec=24.53 ppl=8.779267 [2023-12-17:11:41:19:INFO:sockeye.training:__call__] E=6 B=27750 s/sec=2163.19 tok/sec=107404.48 u/sec=25.37 ppl=8.786765 [2023-12-17:11:41:21:INFO:sockeye.training:__call__] E=6 B=27800 s/sec=2206.30 tok/sec=106224.62 u/sec=25.16 ppl=8.793562 [2023-12-17:11:41:23:INFO:sockeye.training:__call__] E=6 B=27850 s/sec=2340.79 tok/sec=103557.36 u/sec=25.85 ppl=8.803753 [2023-12-17:11:41:25:INFO:sockeye.training:__call__] E=6 B=27900 s/sec=2161.37 tok/sec=108262.03 u/sec=25.20 ppl=8.811251 [2023-12-17:11:41:27:INFO:sockeye.training:__call__] E=6 B=27950 s/sec=2312.03 tok/sec=107631.86 u/sec=25.13 ppl=8.818142 [2023-12-17:11:41:29:INFO:sockeye.training:__call__] E=7 B=28000 s/sec=2181.77 tok/sec=101288.58 u/sec=23.55 ppl=8.809211 [2023-12-17:11:41:29:INFO:sockeye.training:_create_checkpoint] Checkpoint [7] Updates=28000 Epoch=7 Samples=2495344 Time-cost=160.694 Updates/sec=24.892 [2023-12-17:11:41:29:INFO:sockeye.training:_create_checkpoint] Checkpoint [7] Train-ppl=8.809211 [2023-12-17:11:41:32:INFO:sockeye.training:_evaluate] Checkpoint [7] Validation-ppl=28.490051 Validation-bleu=0.154481 Validation-chrf=30.240848 Validation-rouge1=0.154599 Validation-rouge2=0.076176 Validation-rougel=0.146774 Validation-length-ratio-mse=1.323126 Validation-ter=98.803137 Validation-avg-sec-per-sent=0.003546 Validation-decode-walltime=1.773142 [2023-12-17:11:41:32:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 2 checkpoints, best so far: 31.983236 [2023-12-17:11:41:32:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [7] Learning-rate=0.000200 [2023-12-17:11:41:32:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:41:32:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:41:33:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00007" [2023-12-17:11:41:35:INFO:sockeye.training:__call__] E=7 B=28050 s/sec=778.51 tok/sec=36840.06 u/sec=8.98 ppl=7.662700 [2023-12-17:11:41:37:INFO:sockeye.training:__call__] E=7 B=28100 s/sec=2211.97 tok/sec=106780.36 u/sec=24.91 ppl=7.550324 [2023-12-17:11:41:39:INFO:sockeye.training:__call__] E=7 B=28150 s/sec=2128.58 tok/sec=110917.96 u/sec=24.14 ppl=7.537176 [2023-12-17:11:41:41:INFO:sockeye.training:__call__] E=7 B=28200 s/sec=2455.10 tok/sec=107214.13 u/sec=25.32 ppl=7.524609 [2023-12-17:11:41:43:INFO:sockeye.training:__call__] E=7 B=28250 s/sec=2010.43 tok/sec=109608.11 u/sec=24.73 ppl=7.527439 [2023-12-17:11:41:45:INFO:sockeye.training:__call__] E=7 B=28300 s/sec=2077.00 tok/sec=107672.87 u/sec=24.63 ppl=7.534394 [2023-12-17:11:41:47:INFO:sockeye.training:__call__] E=7 B=28350 s/sec=2108.98 tok/sec=105676.06 u/sec=25.35 ppl=7.563868 [2023-12-17:11:41:49:INFO:sockeye.training:__call__] E=7 B=28400 s/sec=2175.56 tok/sec=106678.84 u/sec=25.13 ppl=7.562246 [2023-12-17:11:41:51:INFO:sockeye.training:__call__] E=7 B=28450 s/sec=2221.51 tok/sec=106242.68 u/sec=25.38 ppl=7.573980 [2023-12-17:11:41:53:INFO:sockeye.training:__call__] E=7 B=28500 s/sec=2106.90 tok/sec=96770.58 u/sec=24.16 ppl=7.598141 [2023-12-17:11:41:55:INFO:sockeye.training:__call__] E=7 B=28550 s/sec=2099.75 tok/sec=111523.45 u/sec=24.67 ppl=7.616298 [2023-12-17:11:41:57:INFO:sockeye.training:__call__] E=7 B=28600 s/sec=1970.55 tok/sec=108994.90 u/sec=24.63 ppl=7.622916 [2023-12-17:11:41:59:INFO:sockeye.training:__call__] E=7 B=28650 s/sec=2170.27 tok/sec=109251.36 u/sec=24.80 ppl=7.640320 [2023-12-17:11:42:01:INFO:sockeye.training:__call__] E=7 B=28700 s/sec=2092.16 tok/sec=111229.89 u/sec=24.26 ppl=7.654318 [2023-12-17:11:42:03:INFO:sockeye.training:__call__] E=7 B=28750 s/sec=2339.26 tok/sec=105054.07 u/sec=25.56 ppl=7.669898 [2023-12-17:11:42:05:INFO:sockeye.training:__call__] E=7 B=28800 s/sec=2145.56 tok/sec=108643.99 u/sec=25.11 ppl=7.680952 [2023-12-17:11:42:07:INFO:sockeye.training:__call__] E=7 B=28850 s/sec=2357.91 tok/sec=103308.24 u/sec=25.45 ppl=7.701741 [2023-12-17:11:42:09:INFO:sockeye.training:__call__] E=7 B=28900 s/sec=2160.07 tok/sec=109190.41 u/sec=24.73 ppl=7.720147 [2023-12-17:11:42:11:INFO:sockeye.training:__call__] E=7 B=28950 s/sec=2235.55 tok/sec=107828.78 u/sec=24.01 ppl=7.743600 [2023-12-17:11:42:13:INFO:sockeye.training:__call__] E=7 B=29000 s/sec=2268.12 tok/sec=101089.23 u/sec=23.75 ppl=7.769144 [2023-12-17:11:42:15:INFO:sockeye.training:__call__] E=7 B=29050 s/sec=2340.86 tok/sec=100390.54 u/sec=25.62 ppl=7.781144 [2023-12-17:11:42:17:INFO:sockeye.training:__call__] E=7 B=29100 s/sec=2375.89 tok/sec=105887.40 u/sec=25.38 ppl=7.799289 [2023-12-17:11:42:19:INFO:sockeye.training:__call__] E=7 B=29150 s/sec=2301.85 tok/sec=105747.84 u/sec=25.20 ppl=7.812795 [2023-12-17:11:42:21:INFO:sockeye.training:__call__] E=7 B=29200 s/sec=2049.07 tok/sec=109889.97 u/sec=24.49 ppl=7.824678 [2023-12-17:11:42:23:INFO:sockeye.training:__call__] E=7 B=29250 s/sec=2506.37 tok/sec=101820.06 u/sec=26.02 ppl=7.843846 [2023-12-17:11:42:25:INFO:sockeye.training:__call__] E=7 B=29300 s/sec=2181.63 tok/sec=106260.83 u/sec=25.11 ppl=7.855016 [2023-12-17:11:42:27:INFO:sockeye.training:__call__] E=7 B=29350 s/sec=2324.33 tok/sec=104396.19 u/sec=25.35 ppl=7.868946 [2023-12-17:11:42:29:INFO:sockeye.training:__call__] E=7 B=29400 s/sec=2090.72 tok/sec=107604.62 u/sec=24.84 ppl=7.878014 [2023-12-17:11:42:31:INFO:sockeye.training:__call__] E=7 B=29450 s/sec=2024.98 tok/sec=111744.74 u/sec=24.39 ppl=7.887563 [2023-12-17:11:42:33:INFO:sockeye.training:__call__] E=7 B=29500 s/sec=2190.35 tok/sec=109051.88 u/sec=24.67 ppl=7.902502 [2023-12-17:11:42:35:INFO:sockeye.training:__call__] E=7 B=29550 s/sec=2071.98 tok/sec=112308.96 u/sec=24.43 ppl=7.911449 [2023-12-17:11:42:37:INFO:sockeye.training:__call__] E=7 B=29600 s/sec=2028.51 tok/sec=111436.85 u/sec=24.10 ppl=7.922085 [2023-12-17:11:42:39:INFO:sockeye.training:__call__] E=7 B=29650 s/sec=2523.58 tok/sec=104411.58 u/sec=25.56 ppl=7.939225 [2023-12-17:11:42:41:INFO:sockeye.training:__call__] E=7 B=29700 s/sec=2414.14 tok/sec=106925.77 u/sec=24.98 ppl=7.950084 [2023-12-17:11:42:43:INFO:sockeye.training:__call__] E=7 B=29750 s/sec=2190.88 tok/sec=107723.34 u/sec=24.11 ppl=7.961450 [2023-12-17:11:42:45:INFO:sockeye.training:__call__] E=7 B=29800 s/sec=2199.45 tok/sec=106733.87 u/sec=25.18 ppl=7.972477 [2023-12-17:11:42:47:INFO:sockeye.training:__call__] E=7 B=29850 s/sec=2228.60 tok/sec=107166.34 u/sec=25.19 ppl=7.986896 [2023-12-17:11:42:49:INFO:sockeye.training:__call__] E=7 B=29900 s/sec=2077.33 tok/sec=108782.49 u/sec=24.78 ppl=7.996374 [2023-12-17:11:42:51:INFO:sockeye.training:__call__] E=7 B=29950 s/sec=2326.08 tok/sec=101972.65 u/sec=25.46 ppl=8.013181 [2023-12-17:11:42:53:INFO:sockeye.training:__call__] E=7 B=30000 s/sec=2156.70 tok/sec=104831.11 u/sec=25.58 ppl=8.023223 [2023-12-17:11:42:55:INFO:sockeye.training:__call__] E=7 B=30050 s/sec=2141.83 tok/sec=107067.50 u/sec=24.84 ppl=8.033999 [2023-12-17:11:42:57:INFO:sockeye.training:__call__] E=7 B=30100 s/sec=2158.47 tok/sec=107180.21 u/sec=24.71 ppl=8.044805 [2023-12-17:11:42:59:INFO:sockeye.training:__call__] E=7 B=30150 s/sec=2043.36 tok/sec=108978.95 u/sec=23.52 ppl=8.055748 [2023-12-17:11:43:01:INFO:sockeye.training:__call__] E=7 B=30200 s/sec=2242.99 tok/sec=101286.19 u/sec=24.42 ppl=8.066602 [2023-12-17:11:43:03:INFO:sockeye.training:__call__] E=7 B=30250 s/sec=2399.74 tok/sec=104160.95 u/sec=25.64 ppl=8.079969 [2023-12-17:11:43:05:INFO:sockeye.training:__call__] E=7 B=30300 s/sec=2331.14 tok/sec=104979.36 u/sec=25.61 ppl=8.089466 [2023-12-17:11:43:07:INFO:sockeye.training:__call__] E=7 B=30350 s/sec=2461.79 tok/sec=104971.73 u/sec=25.26 ppl=8.101776 [2023-12-17:11:43:09:INFO:sockeye.training:__call__] E=7 B=30400 s/sec=2498.72 tok/sec=102900.31 u/sec=26.20 ppl=8.114589 [2023-12-17:11:43:11:INFO:sockeye.training:__call__] E=7 B=30450 s/sec=2043.91 tok/sec=110055.74 u/sec=24.06 ppl=8.126053 [2023-12-17:11:43:13:INFO:sockeye.training:__call__] E=7 B=30500 s/sec=1965.38 tok/sec=104777.88 u/sec=24.72 ppl=8.134728 [2023-12-17:11:43:15:INFO:sockeye.training:__call__] E=7 B=30550 s/sec=2413.08 tok/sec=102308.00 u/sec=25.39 ppl=8.148272 [2023-12-17:11:43:17:INFO:sockeye.training:__call__] E=7 B=30600 s/sec=2395.01 tok/sec=103903.97 u/sec=25.63 ppl=8.161414 [2023-12-17:11:43:19:INFO:sockeye.training:__call__] E=7 B=30650 s/sec=2237.37 tok/sec=105242.15 u/sec=25.47 ppl=8.170919 [2023-12-17:11:43:21:INFO:sockeye.training:__call__] E=7 B=30700 s/sec=2390.42 tok/sec=104087.64 u/sec=25.07 ppl=8.181763 [2023-12-17:11:43:23:INFO:sockeye.training:__call__] E=7 B=30750 s/sec=2110.92 tok/sec=106381.23 u/sec=24.39 ppl=8.189242 [2023-12-17:11:43:25:INFO:sockeye.training:__call__] E=7 B=30800 s/sec=2335.60 tok/sec=107670.16 u/sec=24.25 ppl=8.199733 [2023-12-17:11:43:27:INFO:sockeye.training:__call__] E=7 B=30850 s/sec=2411.78 tok/sec=100627.00 u/sec=25.77 ppl=8.208791 [2023-12-17:11:43:29:INFO:sockeye.training:__call__] E=7 B=30900 s/sec=2030.79 tok/sec=110406.14 u/sec=24.46 ppl=8.219238 [2023-12-17:11:43:31:INFO:sockeye.training:__call__] E=7 B=30950 s/sec=2235.90 tok/sec=103210.12 u/sec=24.91 ppl=8.226616 [2023-12-17:11:43:33:INFO:sockeye.training:__call__] E=7 B=31000 s/sec=2376.98 tok/sec=99617.44 u/sec=24.97 ppl=8.234949 [2023-12-17:11:43:35:INFO:sockeye.training:__call__] E=7 B=31050 s/sec=2215.12 tok/sec=105520.05 u/sec=25.17 ppl=8.239767 [2023-12-17:11:43:37:INFO:sockeye.training:__call__] E=7 B=31100 s/sec=2186.14 tok/sec=106270.48 u/sec=24.84 ppl=8.248826 [2023-12-17:11:43:39:INFO:sockeye.training:__call__] E=7 B=31150 s/sec=2155.67 tok/sec=109904.07 u/sec=24.28 ppl=8.257762 [2023-12-17:11:43:41:INFO:sockeye.training:__call__] E=7 B=31200 s/sec=2274.00 tok/sec=103602.45 u/sec=25.65 ppl=8.265491 [2023-12-17:11:43:43:INFO:sockeye.training:__call__] E=7 B=31250 s/sec=2349.64 tok/sec=104309.26 u/sec=25.36 ppl=8.273151 [2023-12-17:11:43:45:INFO:sockeye.training:__call__] E=7 B=31300 s/sec=2107.22 tok/sec=109038.54 u/sec=24.66 ppl=8.280982 [2023-12-17:11:43:47:INFO:sockeye.training:__call__] E=7 B=31350 s/sec=2086.93 tok/sec=112708.90 u/sec=22.88 ppl=8.287824 [2023-12-17:11:43:49:INFO:sockeye.training:__call__] E=7 B=31400 s/sec=2154.79 tok/sec=107331.62 u/sec=24.99 ppl=8.296263 [2023-12-17:11:43:51:INFO:sockeye.training:__call__] E=7 B=31450 s/sec=2278.97 tok/sec=109899.73 u/sec=24.86 ppl=8.303312 [2023-12-17:11:43:54:INFO:sockeye.training:__call__] E=7 B=31500 s/sec=2225.87 tok/sec=110153.04 u/sec=24.41 ppl=8.310775 [2023-12-17:11:43:55:INFO:sockeye.training:__call__] E=7 B=31550 s/sec=2261.96 tok/sec=105960.51 u/sec=25.85 ppl=8.320063 [2023-12-17:11:43:58:INFO:sockeye.training:__call__] E=7 B=31600 s/sec=2155.71 tok/sec=105380.17 u/sec=23.64 ppl=8.327555 [2023-12-17:11:44:00:INFO:sockeye.training:__call__] E=7 B=31650 s/sec=2239.21 tok/sec=106404.25 u/sec=24.77 ppl=8.338708 [2023-12-17:11:44:02:INFO:sockeye.training:__call__] E=7 B=31700 s/sec=2282.55 tok/sec=106064.21 u/sec=25.07 ppl=8.346238 [2023-12-17:11:44:04:INFO:sockeye.training:__call__] E=7 B=31750 s/sec=2315.13 tok/sec=106616.76 u/sec=25.16 ppl=8.357066 [2023-12-17:11:44:06:INFO:sockeye.training:__call__] E=7 B=31800 s/sec=2272.13 tok/sec=105619.74 u/sec=25.13 ppl=8.365381 [2023-12-17:11:44:08:INFO:sockeye.training:__call__] E=7 B=31850 s/sec=2163.21 tok/sec=110178.50 u/sec=24.54 ppl=8.376131 [2023-12-17:11:44:10:INFO:sockeye.training:__call__] E=7 B=31900 s/sec=2142.63 tok/sec=108463.70 u/sec=25.08 ppl=8.382990 [2023-12-17:11:44:12:INFO:sockeye.training:__call__] E=7 B=31950 s/sec=2230.00 tok/sec=104533.68 u/sec=25.76 ppl=8.391453 [2023-12-17:11:44:14:INFO:sockeye.training:__call__] E=8 B=32000 s/sec=2087.27 tok/sec=100337.53 u/sec=23.21 ppl=8.381054 [2023-12-17:11:44:14:INFO:sockeye.training:_create_checkpoint] Checkpoint [8] Updates=32000 Epoch=8 Samples=2851920 Time-cost=160.890 Updates/sec=24.862 [2023-12-17:11:44:14:INFO:sockeye.training:_create_checkpoint] Checkpoint [8] Train-ppl=8.381054 [2023-12-17:11:44:16:INFO:sockeye.training:_evaluate] Checkpoint [8] Validation-ppl=29.687123 Validation-bleu=0.139074 Validation-chrf=29.068812 Validation-rouge1=0.154156 Validation-rouge2=0.072744 Validation-rougel=0.146566 Validation-length-ratio-mse=1.257084 Validation-ter=96.987206 Validation-avg-sec-per-sent=0.003152 Validation-decode-walltime=1.575938 [2023-12-17:11:44:16:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 3 checkpoints, best so far: 31.983236 [2023-12-17:11:44:16:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [8] Learning-rate=0.000200 [2023-12-17:11:44:17:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:44:17:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=3, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:44:17:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00008" [2023-12-17:11:44:19:INFO:sockeye.training:__call__] E=8 B=32050 s/sec=904.51 tok/sec=38369.44 u/sec=9.50 ppl=7.374685 [2023-12-17:11:44:21:INFO:sockeye.training:__call__] E=8 B=32100 s/sec=2267.83 tok/sec=107134.28 u/sec=24.91 ppl=7.284335 [2023-12-17:11:44:23:INFO:sockeye.training:__call__] E=8 B=32150 s/sec=2295.09 tok/sec=104219.32 u/sec=25.66 ppl=7.288698 [2023-12-17:11:44:25:INFO:sockeye.training:__call__] E=8 B=32200 s/sec=2056.30 tok/sec=109596.87 u/sec=24.25 ppl=7.212497 [2023-12-17:11:44:27:INFO:sockeye.training:__call__] E=8 B=32250 s/sec=2095.02 tok/sec=108320.70 u/sec=24.85 ppl=7.241880 [2023-12-17:11:44:29:INFO:sockeye.training:__call__] E=8 B=32300 s/sec=2178.49 tok/sec=110585.90 u/sec=24.49 ppl=7.262406 [2023-12-17:11:44:31:INFO:sockeye.training:__call__] E=8 B=32350 s/sec=2231.80 tok/sec=105494.31 u/sec=25.50 ppl=7.276950 [2023-12-17:11:44:33:INFO:sockeye.training:__call__] E=8 B=32400 s/sec=2296.81 tok/sec=106663.31 u/sec=25.45 ppl=7.278899 [2023-12-17:11:44:35:INFO:sockeye.training:__call__] E=8 B=32450 s/sec=2167.74 tok/sec=108274.34 u/sec=25.18 ppl=7.285471 [2023-12-17:11:44:37:INFO:sockeye.training:__call__] E=8 B=32500 s/sec=2442.48 tok/sec=104201.35 u/sec=25.66 ppl=7.309207 [2023-12-17:11:44:39:INFO:sockeye.training:__call__] E=8 B=32550 s/sec=2337.99 tok/sec=103460.21 u/sec=25.55 ppl=7.329518 [2023-12-17:11:44:41:INFO:sockeye.training:__call__] E=8 B=32600 s/sec=1990.31 tok/sec=109944.08 u/sec=24.63 ppl=7.342921 [2023-12-17:11:44:43:INFO:sockeye.training:__call__] E=8 B=32650 s/sec=2401.68 tok/sec=108222.06 u/sec=24.73 ppl=7.367640 [2023-12-17:11:44:45:INFO:sockeye.training:__call__] E=8 B=32700 s/sec=2366.26 tok/sec=105747.18 u/sec=25.50 ppl=7.377775 [2023-12-17:11:44:47:INFO:sockeye.training:__call__] E=8 B=32750 s/sec=2181.57 tok/sec=105101.57 u/sec=25.16 ppl=7.391007 [2023-12-17:11:44:49:INFO:sockeye.training:__call__] E=8 B=32800 s/sec=2129.06 tok/sec=106388.43 u/sec=25.20 ppl=7.398178 [2023-12-17:11:44:51:INFO:sockeye.training:__call__] E=8 B=32850 s/sec=2088.63 tok/sec=110558.15 u/sec=24.17 ppl=7.414154 [2023-12-17:11:44:53:INFO:sockeye.training:__call__] E=8 B=32900 s/sec=2449.00 tok/sec=105126.15 u/sec=25.68 ppl=7.433600 [2023-12-17:11:44:55:INFO:sockeye.training:__call__] E=8 B=32950 s/sec=1958.42 tok/sec=119892.10 u/sec=22.88 ppl=7.445136 [2023-12-17:11:44:57:INFO:sockeye.training:__call__] E=8 B=33000 s/sec=2189.52 tok/sec=110310.37 u/sec=24.84 ppl=7.452323 [2023-12-17:11:44:59:INFO:sockeye.training:__call__] E=8 B=33050 s/sec=2209.09 tok/sec=106817.64 u/sec=25.43 ppl=7.468690 [2023-12-17:11:45:01:INFO:sockeye.training:__call__] E=8 B=33100 s/sec=2160.02 tok/sec=111180.61 u/sec=24.41 ppl=7.477354 [2023-12-17:11:45:03:INFO:sockeye.training:__call__] E=8 B=33150 s/sec=2047.99 tok/sec=110771.34 u/sec=24.47 ppl=7.485241 [2023-12-17:11:45:05:INFO:sockeye.training:__call__] E=8 B=33200 s/sec=2116.29 tok/sec=109556.66 u/sec=24.72 ppl=7.500502 [2023-12-17:11:45:07:INFO:sockeye.training:__call__] E=8 B=33250 s/sec=2124.42 tok/sec=109008.81 u/sec=24.54 ppl=7.512975 [2023-12-17:11:45:09:INFO:sockeye.training:__call__] E=8 B=33300 s/sec=2217.95 tok/sec=108068.88 u/sec=25.11 ppl=7.526637 [2023-12-17:11:45:11:INFO:sockeye.training:__call__] E=8 B=33350 s/sec=2057.30 tok/sec=110809.42 u/sec=24.73 ppl=7.537450 [2023-12-17:11:45:13:INFO:sockeye.training:__call__] E=8 B=33400 s/sec=2472.22 tok/sec=101215.46 u/sec=25.97 ppl=7.550631 [2023-12-17:11:45:15:INFO:sockeye.training:__call__] E=8 B=33450 s/sec=2194.30 tok/sec=105654.42 u/sec=25.63 ppl=7.558947 [2023-12-17:11:45:17:INFO:sockeye.training:__call__] E=8 B=33500 s/sec=2025.82 tok/sec=110047.41 u/sec=24.59 ppl=7.567783 [2023-12-17:11:45:19:INFO:sockeye.training:__call__] E=8 B=33550 s/sec=2144.01 tok/sec=108352.45 u/sec=25.00 ppl=7.573907 [2023-12-17:11:45:21:INFO:sockeye.training:__call__] E=8 B=33600 s/sec=2141.93 tok/sec=109799.75 u/sec=24.70 ppl=7.577943 [2023-12-17:11:45:23:INFO:sockeye.training:__call__] E=8 B=33650 s/sec=2480.21 tok/sec=101898.99 u/sec=26.27 ppl=7.593129 [2023-12-17:11:45:25:INFO:sockeye.training:__call__] E=8 B=33700 s/sec=2355.58 tok/sec=103904.53 u/sec=25.30 ppl=7.610253 [2023-12-17:11:45:27:INFO:sockeye.training:__call__] E=8 B=33750 s/sec=2411.25 tok/sec=104787.41 u/sec=25.54 ppl=7.623166 [2023-12-17:11:45:29:INFO:sockeye.training:__call__] E=8 B=33800 s/sec=2325.87 tok/sec=105769.10 u/sec=24.93 ppl=7.636165 [2023-12-17:11:45:31:INFO:sockeye.training:__call__] E=8 B=33850 s/sec=2141.21 tok/sec=107371.48 u/sec=24.92 ppl=7.642625 [2023-12-17:11:45:33:INFO:sockeye.training:__call__] E=8 B=33900 s/sec=2343.73 tok/sec=104398.79 u/sec=25.21 ppl=7.653104 [2023-12-17:11:45:35:INFO:sockeye.training:__call__] E=8 B=33950 s/sec=2221.58 tok/sec=108114.38 u/sec=24.58 ppl=7.666198 [2023-12-17:11:45:37:INFO:sockeye.training:__call__] E=8 B=34000 s/sec=2121.72 tok/sec=110876.61 u/sec=24.42 ppl=7.675919 [2023-12-17:11:45:39:INFO:sockeye.training:__call__] E=8 B=34050 s/sec=1983.32 tok/sec=109896.67 u/sec=24.69 ppl=7.685378 [2023-12-17:11:45:41:INFO:sockeye.training:__call__] E=8 B=34100 s/sec=2325.83 tok/sec=106186.16 u/sec=25.06 ppl=7.694672 [2023-12-17:11:45:43:INFO:sockeye.training:__call__] E=8 B=34150 s/sec=2513.55 tok/sec=104013.65 u/sec=25.59 ppl=7.710421 [2023-12-17:11:45:45:INFO:sockeye.training:__call__] E=8 B=34200 s/sec=2174.26 tok/sec=106538.89 u/sec=24.98 ppl=7.719820 [2023-12-17:11:45:47:INFO:sockeye.training:__call__] E=8 B=34250 s/sec=2172.42 tok/sec=108754.97 u/sec=24.64 ppl=7.728479 [2023-12-17:11:45:49:INFO:sockeye.training:__call__] E=8 B=34300 s/sec=2302.98 tok/sec=105584.08 u/sec=25.66 ppl=7.741052 [2023-12-17:11:45:51:INFO:sockeye.training:__call__] E=8 B=34350 s/sec=2273.91 tok/sec=106404.34 u/sec=25.52 ppl=7.753390 [2023-12-17:11:45:53:INFO:sockeye.training:__call__] E=8 B=34400 s/sec=2150.00 tok/sec=109397.30 u/sec=24.70 ppl=7.764252 [2023-12-17:11:45:55:INFO:sockeye.training:__call__] E=8 B=34450 s/sec=2318.11 tok/sec=105618.26 u/sec=25.37 ppl=7.774273 [2023-12-17:11:45:57:INFO:sockeye.training:__call__] E=8 B=34500 s/sec=2323.87 tok/sec=106451.95 u/sec=25.35 ppl=7.786409 [2023-12-17:11:45:59:INFO:sockeye.training:__call__] E=8 B=34550 s/sec=1986.10 tok/sec=109641.89 u/sec=24.20 ppl=7.796474 [2023-12-17:11:46:01:INFO:sockeye.training:__call__] E=8 B=34600 s/sec=2404.93 tok/sec=105319.11 u/sec=25.09 ppl=7.807753 [2023-12-17:11:46:03:INFO:sockeye.training:__call__] E=8 B=34650 s/sec=2483.78 tok/sec=105513.31 u/sec=25.74 ppl=7.818553 [2023-12-17:11:46:05:INFO:sockeye.training:__call__] E=8 B=34700 s/sec=2306.19 tok/sec=104849.07 u/sec=25.24 ppl=7.828945 [2023-12-17:11:46:07:INFO:sockeye.training:__call__] E=8 B=34750 s/sec=2227.28 tok/sec=104957.39 u/sec=24.59 ppl=7.835011 [2023-12-17:11:46:09:INFO:sockeye.training:__call__] E=8 B=34800 s/sec=2247.54 tok/sec=108937.85 u/sec=24.99 ppl=7.844035 [2023-12-17:11:46:11:INFO:sockeye.training:__call__] E=8 B=34850 s/sec=2300.64 tok/sec=104107.94 u/sec=25.86 ppl=7.856753 [2023-12-17:11:46:13:INFO:sockeye.training:__call__] E=8 B=34900 s/sec=2259.84 tok/sec=104210.78 u/sec=26.01 ppl=7.866614 [2023-12-17:11:46:15:INFO:sockeye.training:__call__] E=8 B=34950 s/sec=2246.56 tok/sec=109018.07 u/sec=24.98 ppl=7.876000 [2023-12-17:11:46:17:INFO:sockeye.training:__call__] E=8 B=35000 s/sec=2252.44 tok/sec=106056.97 u/sec=25.55 ppl=7.883852 [2023-12-17:11:46:19:INFO:sockeye.training:__call__] E=8 B=35050 s/sec=2236.60 tok/sec=109394.21 u/sec=24.87 ppl=7.891437 [2023-12-17:11:46:21:INFO:sockeye.training:__call__] E=8 B=35100 s/sec=2149.32 tok/sec=109725.37 u/sec=24.51 ppl=7.896862 [2023-12-17:11:46:23:INFO:sockeye.training:__call__] E=8 B=35150 s/sec=2363.24 tok/sec=107357.47 u/sec=24.99 ppl=7.908335 [2023-12-17:11:46:25:INFO:sockeye.training:__call__] E=8 B=35200 s/sec=2072.21 tok/sec=109437.79 u/sec=24.58 ppl=7.915861 [2023-12-17:11:46:27:INFO:sockeye.training:__call__] E=8 B=35250 s/sec=2115.16 tok/sec=109511.43 u/sec=24.85 ppl=7.921853 [2023-12-17:11:46:29:INFO:sockeye.training:__call__] E=8 B=35300 s/sec=2449.75 tok/sec=103281.35 u/sec=25.52 ppl=7.932048 [2023-12-17:11:46:31:INFO:sockeye.training:__call__] E=8 B=35350 s/sec=1971.63 tok/sec=110537.37 u/sec=24.65 ppl=7.937140 [2023-12-17:11:46:33:INFO:sockeye.training:__call__] E=8 B=35400 s/sec=2333.34 tok/sec=105895.75 u/sec=25.10 ppl=7.944150 [2023-12-17:11:46:35:INFO:sockeye.training:__call__] E=8 B=35450 s/sec=2463.28 tok/sec=101565.62 u/sec=26.05 ppl=7.956362 [2023-12-17:11:46:37:INFO:sockeye.training:__call__] E=8 B=35500 s/sec=2398.71 tok/sec=101852.95 u/sec=25.63 ppl=7.967422 [2023-12-17:11:46:39:INFO:sockeye.training:__call__] E=8 B=35550 s/sec=1936.18 tok/sec=107694.40 u/sec=24.85 ppl=7.973260 [2023-12-17:11:46:41:INFO:sockeye.training:__call__] E=8 B=35600 s/sec=2098.54 tok/sec=109266.52 u/sec=24.70 ppl=7.979335 [2023-12-17:11:46:43:INFO:sockeye.training:__call__] E=8 B=35650 s/sec=2445.85 tok/sec=104683.84 u/sec=25.61 ppl=7.988933 [2023-12-17:11:46:45:INFO:sockeye.training:__call__] E=8 B=35700 s/sec=2358.00 tok/sec=105471.18 u/sec=25.11 ppl=7.996464 [2023-12-17:11:46:47:INFO:sockeye.training:__call__] E=8 B=35750 s/sec=2317.17 tok/sec=104654.77 u/sec=25.41 ppl=8.004386 [2023-12-17:11:46:49:INFO:sockeye.training:__call__] E=8 B=35800 s/sec=2026.88 tok/sec=106312.72 u/sec=25.09 ppl=8.011009 [2023-12-17:11:46:51:INFO:sockeye.training:__call__] E=8 B=35850 s/sec=2020.28 tok/sec=109798.22 u/sec=24.14 ppl=8.017231 [2023-12-17:11:46:53:INFO:sockeye.training:__call__] E=8 B=35900 s/sec=2369.34 tok/sec=104423.92 u/sec=24.56 ppl=8.026023 [2023-12-17:11:46:55:INFO:sockeye.training:__call__] E=8 B=35950 s/sec=2265.89 tok/sec=107645.57 u/sec=24.93 ppl=8.036550 [2023-12-17:11:46:57:INFO:sockeye.training:__call__] E=9 B=36000 s/sec=1890.98 tok/sec=105238.49 u/sec=23.31 ppl=8.023116 [2023-12-17:11:46:57:INFO:sockeye.training:_create_checkpoint] Checkpoint [9] Updates=36000 Epoch=9 Samples=3208040 Time-cost=159.915 Updates/sec=25.013 [2023-12-17:11:46:57:INFO:sockeye.training:_create_checkpoint] Checkpoint [9] Train-ppl=8.023116 [2023-12-17:11:46:59:INFO:sockeye.training:_evaluate] Checkpoint [9] Validation-ppl=29.884377 Validation-bleu=0.165922 Validation-chrf=31.158840 Validation-rouge1=0.163998 Validation-rouge2=0.081888 Validation-rougel=0.156817 Validation-length-ratio-mse=1.333691 Validation-ter=97.606273 Validation-avg-sec-per-sent=0.003181 Validation-decode-walltime=1.590617 [2023-12-17:11:46:59:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 4 checkpoints, best so far: 31.983236 [2023-12-17:11:46:59:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [9] Learning-rate=0.000200 [2023-12-17:11:47:00:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:47:00:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=4, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:47:00:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00009" [2023-12-17:11:47:02:INFO:sockeye.training:__call__] E=9 B=36050 s/sec=804.87 tok/sec=40009.21 u/sec=9.26 ppl=6.901913 [2023-12-17:11:47:04:INFO:sockeye.training:__call__] E=9 B=36100 s/sec=1918.08 tok/sec=111871.11 u/sec=24.02 ppl=6.890963 [2023-12-17:11:47:06:INFO:sockeye.training:__call__] E=9 B=36150 s/sec=2521.67 tok/sec=103196.99 u/sec=26.01 ppl=6.962972 [2023-12-17:11:47:08:INFO:sockeye.training:__call__] E=9 B=36200 s/sec=2211.00 tok/sec=107713.08 u/sec=24.76 ppl=6.987483 [2023-12-17:11:47:10:INFO:sockeye.training:__call__] E=9 B=36250 s/sec=2061.33 tok/sec=109672.06 u/sec=24.87 ppl=6.986610 [2023-12-17:11:47:12:INFO:sockeye.training:__call__] E=9 B=36300 s/sec=2334.34 tok/sec=106721.82 u/sec=24.77 ppl=6.997938 [2023-12-17:11:47:14:INFO:sockeye.training:__call__] E=9 B=36350 s/sec=2279.76 tok/sec=109176.20 u/sec=24.65 ppl=7.018161 [2023-12-17:11:47:16:INFO:sockeye.training:__call__] E=9 B=36400 s/sec=2295.83 tok/sec=105931.33 u/sec=25.26 ppl=7.040086 [2023-12-17:11:47:18:INFO:sockeye.training:__call__] E=9 B=36450 s/sec=2024.88 tok/sec=110582.67 u/sec=24.67 ppl=7.045817 [2023-12-17:11:47:20:INFO:sockeye.training:__call__] E=9 B=36500 s/sec=2250.63 tok/sec=104854.86 u/sec=24.81 ppl=7.068834 [2023-12-17:11:47:22:INFO:sockeye.training:__call__] E=9 B=36550 s/sec=2231.85 tok/sec=108862.69 u/sec=25.09 ppl=7.084538 [2023-12-17:11:47:24:INFO:sockeye.training:__call__] E=9 B=36600 s/sec=2358.45 tok/sec=102938.12 u/sec=25.55 ppl=7.107569 [2023-12-17:11:47:26:INFO:sockeye.training:__call__] E=9 B=36650 s/sec=2292.74 tok/sec=105319.04 u/sec=25.50 ppl=7.120186 [2023-12-17:11:47:28:INFO:sockeye.training:__call__] E=9 B=36700 s/sec=2274.71 tok/sec=106003.28 u/sec=25.12 ppl=7.139070 [2023-12-17:11:47:30:INFO:sockeye.training:__call__] E=9 B=36750 s/sec=2050.33 tok/sec=109520.30 u/sec=24.79 ppl=7.154487 [2023-12-17:11:47:32:INFO:sockeye.training:__call__] E=9 B=36800 s/sec=2099.15 tok/sec=110676.44 u/sec=24.75 ppl=7.163981 [2023-12-17:11:47:34:INFO:sockeye.training:__call__] E=9 B=36850 s/sec=2258.85 tok/sec=107584.26 u/sec=25.26 ppl=7.171898 [2023-12-17:11:47:36:INFO:sockeye.training:__call__] E=9 B=36900 s/sec=2175.86 tok/sec=105101.32 u/sec=24.55 ppl=7.193922 [2023-12-17:11:47:38:INFO:sockeye.training:__call__] E=9 B=36950 s/sec=2301.79 tok/sec=105034.36 u/sec=24.76 ppl=7.203685 [2023-12-17:11:47:40:INFO:sockeye.training:__call__] E=9 B=37000 s/sec=1949.20 tok/sec=108176.69 u/sec=24.66 ppl=7.204760 [2023-12-17:11:47:42:INFO:sockeye.training:__call__] E=9 B=37050 s/sec=2080.06 tok/sec=110415.01 u/sec=24.44 ppl=7.213642 [2023-12-17:11:47:44:INFO:sockeye.training:__call__] E=9 B=37100 s/sec=2277.82 tok/sec=107314.16 u/sec=25.06 ppl=7.221283 [2023-12-17:11:47:46:INFO:sockeye.training:__call__] E=9 B=37150 s/sec=2176.71 tok/sec=109835.47 u/sec=24.60 ppl=7.230839 [2023-12-17:11:47:48:INFO:sockeye.training:__call__] E=9 B=37200 s/sec=2229.33 tok/sec=108800.93 u/sec=25.01 ppl=7.239676 [2023-12-17:11:47:51:INFO:sockeye.training:__call__] E=9 B=37250 s/sec=1930.66 tok/sec=112233.57 u/sec=24.18 ppl=7.240490 [2023-12-17:11:47:53:INFO:sockeye.training:__call__] E=9 B=37300 s/sec=2182.48 tok/sec=105471.51 u/sec=25.31 ppl=7.249252 [2023-12-17:11:47:55:INFO:sockeye.training:__call__] E=9 B=37350 s/sec=2165.72 tok/sec=108776.38 u/sec=25.11 ppl=7.262040 [2023-12-17:11:47:57:INFO:sockeye.training:__call__] E=9 B=37400 s/sec=2287.23 tok/sec=106699.70 u/sec=25.12 ppl=7.274979 [2023-12-17:11:47:58:INFO:sockeye.training:__call__] E=9 B=37450 s/sec=2381.93 tok/sec=102971.46 u/sec=25.58 ppl=7.286144 [2023-12-17:11:48:00:INFO:sockeye.training:__call__] E=9 B=37500 s/sec=2291.64 tok/sec=103810.81 u/sec=25.71 ppl=7.299672 [2023-12-17:11:48:02:INFO:sockeye.training:__call__] E=9 B=37550 s/sec=2341.87 tok/sec=104138.47 u/sec=25.95 ppl=7.312457 [2023-12-17:11:48:04:INFO:sockeye.training:__call__] E=9 B=37600 s/sec=2119.28 tok/sec=114961.77 u/sec=23.24 ppl=7.321501 [2023-12-17:11:48:06:INFO:sockeye.training:__call__] E=9 B=37650 s/sec=2268.67 tok/sec=105572.61 u/sec=25.46 ppl=7.333908 [2023-12-17:11:48:08:INFO:sockeye.training:__call__] E=9 B=37700 s/sec=2430.13 tok/sec=102977.75 u/sec=25.92 ppl=7.347161 [2023-12-17:11:48:10:INFO:sockeye.training:__call__] E=9 B=37750 s/sec=2179.97 tok/sec=109435.42 u/sec=24.82 ppl=7.359585 [2023-12-17:11:48:12:INFO:sockeye.training:__call__] E=9 B=37800 s/sec=2404.58 tok/sec=104002.11 u/sec=25.56 ppl=7.370550 [2023-12-17:11:48:14:INFO:sockeye.training:__call__] E=9 B=37850 s/sec=1930.10 tok/sec=110004.09 u/sec=24.57 ppl=7.374790 [2023-12-17:11:48:16:INFO:sockeye.training:__call__] E=9 B=37900 s/sec=2296.28 tok/sec=104899.50 u/sec=25.05 ppl=7.385159 [2023-12-17:11:48:18:INFO:sockeye.training:__call__] E=9 B=37950 s/sec=2380.82 tok/sec=105345.32 u/sec=25.57 ppl=7.396392 [2023-12-17:11:48:20:INFO:sockeye.training:__call__] E=9 B=38000 s/sec=2183.59 tok/sec=107326.22 u/sec=24.86 ppl=7.408460 [2023-12-17:11:48:22:INFO:sockeye.training:__call__] E=9 B=38050 s/sec=2076.09 tok/sec=107612.16 u/sec=24.48 ppl=7.417181 [2023-12-17:11:48:24:INFO:sockeye.training:__call__] E=9 B=38100 s/sec=2421.80 tok/sec=102769.94 u/sec=25.74 ppl=7.426840 [2023-12-17:11:48:26:INFO:sockeye.training:__call__] E=9 B=38150 s/sec=2594.11 tok/sec=101949.91 u/sec=25.78 ppl=7.436822 [2023-12-17:11:48:28:INFO:sockeye.training:__call__] E=9 B=38200 s/sec=2216.76 tok/sec=109439.87 u/sec=24.96 ppl=7.450195 [2023-12-17:11:48:30:INFO:sockeye.training:__call__] E=9 B=38250 s/sec=2021.46 tok/sec=112272.12 u/sec=24.20 ppl=7.455019 [2023-12-17:11:48:32:INFO:sockeye.training:__call__] E=9 B=38300 s/sec=2325.24 tok/sec=102817.09 u/sec=25.54 ppl=7.465732 [2023-12-17:11:48:34:INFO:sockeye.training:__call__] E=9 B=38350 s/sec=2479.12 tok/sec=102399.34 u/sec=25.53 ppl=7.477238 [2023-12-17:11:48:36:INFO:sockeye.training:__call__] E=9 B=38400 s/sec=2183.74 tok/sec=106348.50 u/sec=25.42 ppl=7.483660 [2023-12-17:11:48:38:INFO:sockeye.training:__call__] E=9 B=38450 s/sec=2188.70 tok/sec=108941.31 u/sec=24.87 ppl=7.491617 [2023-12-17:11:48:40:INFO:sockeye.training:__call__] E=9 B=38500 s/sec=2356.10 tok/sec=105437.58 u/sec=25.13 ppl=7.501322 [2023-12-17:11:48:42:INFO:sockeye.training:__call__] E=9 B=38550 s/sec=2459.93 tok/sec=106134.22 u/sec=25.08 ppl=7.511101 [2023-12-17:11:48:44:INFO:sockeye.training:__call__] E=9 B=38600 s/sec=1945.89 tok/sec=111206.86 u/sec=24.28 ppl=7.518937 [2023-12-17:11:48:46:INFO:sockeye.training:__call__] E=9 B=38650 s/sec=2270.30 tok/sec=106759.41 u/sec=24.72 ppl=7.530249 [2023-12-17:11:48:48:INFO:sockeye.training:__call__] E=9 B=38700 s/sec=2024.03 tok/sec=108620.30 u/sec=24.61 ppl=7.537113 [2023-12-17:11:48:50:INFO:sockeye.training:__call__] E=9 B=38750 s/sec=2343.72 tok/sec=104005.55 u/sec=25.52 ppl=7.546878 [2023-12-17:11:48:52:INFO:sockeye.training:__call__] E=9 B=38800 s/sec=2523.91 tok/sec=103451.79 u/sec=25.65 ppl=7.556851 [2023-12-17:11:48:54:INFO:sockeye.training:__call__] E=9 B=38850 s/sec=2203.42 tok/sec=108607.26 u/sec=24.68 ppl=7.564461 [2023-12-17:11:48:56:INFO:sockeye.training:__call__] E=9 B=38900 s/sec=2237.76 tok/sec=106353.87 u/sec=25.06 ppl=7.572251 [2023-12-17:11:48:58:INFO:sockeye.training:__call__] E=9 B=38950 s/sec=2187.94 tok/sec=107045.91 u/sec=25.51 ppl=7.581315 [2023-12-17:11:49:00:INFO:sockeye.training:__call__] E=9 B=39000 s/sec=2284.80 tok/sec=106532.38 u/sec=25.27 ppl=7.591142 [2023-12-17:11:49:02:INFO:sockeye.training:__call__] E=9 B=39050 s/sec=2252.92 tok/sec=103937.90 u/sec=25.65 ppl=7.599432 [2023-12-17:11:49:04:INFO:sockeye.training:__call__] E=9 B=39100 s/sec=2527.24 tok/sec=102657.51 u/sec=26.06 ppl=7.610139 [2023-12-17:11:49:06:INFO:sockeye.training:__call__] E=9 B=39150 s/sec=2325.67 tok/sec=103973.11 u/sec=25.68 ppl=7.618956 [2023-12-17:11:49:08:INFO:sockeye.training:__call__] E=9 B=39200 s/sec=2369.86 tok/sec=107046.90 u/sec=25.19 ppl=7.629259 [2023-12-17:11:49:10:INFO:sockeye.training:__call__] E=9 B=39250 s/sec=2429.04 tok/sec=104496.90 u/sec=25.13 ppl=7.638338 [2023-12-17:11:49:12:INFO:sockeye.training:__call__] E=9 B=39300 s/sec=1991.37 tok/sec=109195.83 u/sec=24.50 ppl=7.646165 [2023-12-17:11:49:14:INFO:sockeye.training:__call__] E=9 B=39350 s/sec=1785.10 tok/sec=113973.02 u/sec=23.74 ppl=7.649940 [2023-12-17:11:49:16:INFO:sockeye.training:__call__] E=9 B=39400 s/sec=2226.87 tok/sec=108456.88 u/sec=24.99 ppl=7.657611 [2023-12-17:11:49:18:INFO:sockeye.training:__call__] E=9 B=39450 s/sec=2240.43 tok/sec=109903.20 u/sec=24.61 ppl=7.664642 [2023-12-17:11:49:20:INFO:sockeye.training:__call__] E=9 B=39500 s/sec=2063.90 tok/sec=107258.41 u/sec=25.19 ppl=7.671282 [2023-12-17:11:49:22:INFO:sockeye.training:__call__] E=9 B=39550 s/sec=2350.61 tok/sec=107040.84 u/sec=24.99 ppl=7.680298 [2023-12-17:11:49:24:INFO:sockeye.training:__call__] E=9 B=39600 s/sec=2402.37 tok/sec=106172.49 u/sec=25.23 ppl=7.690274 [2023-12-17:11:49:26:INFO:sockeye.training:__call__] E=9 B=39650 s/sec=2367.51 tok/sec=105469.09 u/sec=24.74 ppl=7.697810 [2023-12-17:11:49:28:INFO:sockeye.training:__call__] E=9 B=39700 s/sec=2382.51 tok/sec=104879.14 u/sec=25.28 ppl=7.706471 [2023-12-17:11:49:30:INFO:sockeye.training:__call__] E=9 B=39750 s/sec=2196.58 tok/sec=106592.26 u/sec=25.10 ppl=7.714518 [2023-12-17:11:49:32:INFO:sockeye.training:__call__] E=9 B=39800 s/sec=2176.05 tok/sec=104803.78 u/sec=25.09 ppl=7.720561 [2023-12-17:11:49:34:INFO:sockeye.training:__call__] E=9 B=39850 s/sec=2002.38 tok/sec=109514.86 u/sec=24.07 ppl=7.726653 [2023-12-17:11:49:36:INFO:sockeye.training:__call__] E=9 B=39900 s/sec=2230.10 tok/sec=106205.98 u/sec=25.20 ppl=7.734975 [2023-12-17:11:49:38:INFO:sockeye.training:__call__] E=9 B=39950 s/sec=2268.57 tok/sec=106027.69 u/sec=25.14 ppl=7.743428 [2023-12-17:11:49:40:INFO:sockeye.training:__call__] E=10 B=40000 s/sec=2167.39 tok/sec=102367.64 u/sec=23.68 ppl=7.731083 [2023-12-17:11:49:40:INFO:sockeye.training:_create_checkpoint] Checkpoint [10] Updates=40000 Epoch=10 Samples=3564976 Time-cost=160.041 Updates/sec=24.994 [2023-12-17:11:49:40:INFO:sockeye.training:_create_checkpoint] Checkpoint [10] Train-ppl=7.731083 [2023-12-17:11:49:43:INFO:sockeye.training:_evaluate] Checkpoint [10] Validation-ppl=31.639580 Validation-bleu=0.172840 Validation-chrf=31.113908 Validation-rouge1=0.156950 Validation-rouge2=0.075279 Validation-rougel=0.149662 Validation-length-ratio-mse=1.303005 Validation-ter=96.657037 Validation-avg-sec-per-sent=0.003251 Validation-decode-walltime=1.625525 [2023-12-17:11:49:43:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 5 checkpoints, best so far: 31.983236 [2023-12-17:11:49:43:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [10] Learning-rate=0.000200 [2023-12-17:11:49:43:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:49:43:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=5, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:49:44:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00010" [2023-12-17:11:49:46:INFO:sockeye.training:__call__] E=10 B=40050 s/sec=837.17 tok/sec=41102.51 u/sec=9.57 ppl=6.775582 [2023-12-17:11:49:48:INFO:sockeye.training:__call__] E=10 B=40100 s/sec=2259.61 tok/sec=105145.73 u/sec=24.91 ppl=6.785598 [2023-12-17:11:49:50:INFO:sockeye.training:__call__] E=10 B=40150 s/sec=2512.10 tok/sec=103008.44 u/sec=25.28 ppl=6.823666 [2023-12-17:11:49:52:INFO:sockeye.training:__call__] E=10 B=40200 s/sec=2147.83 tok/sec=109108.35 u/sec=24.72 ppl=6.822364 [2023-12-17:11:49:54:INFO:sockeye.training:__call__] E=10 B=40250 s/sec=1909.03 tok/sec=110229.76 u/sec=24.50 ppl=6.801383 [2023-12-17:11:49:56:INFO:sockeye.training:__call__] E=10 B=40300 s/sec=2495.55 tok/sec=101491.24 u/sec=25.57 ppl=6.826864 [2023-12-17:11:49:58:INFO:sockeye.training:__call__] E=10 B=40350 s/sec=2029.13 tok/sec=110535.03 u/sec=24.44 ppl=6.819743 [2023-12-17:11:50:00:INFO:sockeye.training:__call__] E=10 B=40400 s/sec=2295.18 tok/sec=104650.31 u/sec=25.21 ppl=6.832038 [2023-12-17:11:50:02:INFO:sockeye.training:__call__] E=10 B=40450 s/sec=2020.73 tok/sec=110445.48 u/sec=24.67 ppl=6.834407 [2023-12-17:11:50:04:INFO:sockeye.training:__call__] E=10 B=40500 s/sec=2132.76 tok/sec=109115.79 u/sec=24.50 ppl=6.837348 [2023-12-17:11:50:06:INFO:sockeye.training:__call__] E=10 B=40550 s/sec=2284.36 tok/sec=102692.01 u/sec=26.01 ppl=6.851905 [2023-12-17:11:50:08:INFO:sockeye.training:__call__] E=10 B=40600 s/sec=2266.64 tok/sec=108120.35 u/sec=24.09 ppl=6.858550 [2023-12-17:11:50:10:INFO:sockeye.training:__call__] E=10 B=40650 s/sec=2016.86 tok/sec=106503.09 u/sec=25.16 ppl=6.869249 [2023-12-17:11:50:12:INFO:sockeye.training:__call__] E=10 B=40700 s/sec=2193.91 tok/sec=106146.51 u/sec=25.21 ppl=6.885398 [2023-12-17:11:50:14:INFO:sockeye.training:__call__] E=10 B=40750 s/sec=1827.11 tok/sec=109260.53 u/sec=24.88 ppl=6.884065 [2023-12-17:11:50:16:INFO:sockeye.training:__call__] E=10 B=40800 s/sec=2209.38 tok/sec=105798.39 u/sec=24.61 ppl=6.901469 [2023-12-17:11:50:18:INFO:sockeye.training:__call__] E=10 B=40850 s/sec=2615.25 tok/sec=103104.01 u/sec=25.58 ppl=6.919566 [2023-12-17:11:50:20:INFO:sockeye.training:__call__] E=10 B=40900 s/sec=2322.61 tok/sec=102293.76 u/sec=25.74 ppl=6.931080 [2023-12-17:11:50:22:INFO:sockeye.training:__call__] E=10 B=40950 s/sec=2370.52 tok/sec=102276.62 u/sec=25.33 ppl=6.947948 [2023-12-17:11:50:24:INFO:sockeye.training:__call__] E=10 B=41000 s/sec=2161.95 tok/sec=109209.36 u/sec=24.13 ppl=6.960431 [2023-12-17:11:50:26:INFO:sockeye.training:__call__] E=10 B=41050 s/sec=2031.92 tok/sec=108379.77 u/sec=24.56 ppl=6.964306 [2023-12-17:11:50:28:INFO:sockeye.training:__call__] E=10 B=41100 s/sec=2362.53 tok/sec=103951.39 u/sec=25.46 ppl=6.983764 [2023-12-17:11:50:30:INFO:sockeye.training:__call__] E=10 B=41150 s/sec=2187.38 tok/sec=109330.22 u/sec=24.37 ppl=6.996544 [2023-12-17:11:50:32:INFO:sockeye.training:__call__] E=10 B=41200 s/sec=2695.85 tok/sec=99274.05 u/sec=26.12 ppl=7.018462 [2023-12-17:11:50:34:INFO:sockeye.training:__call__] E=10 B=41250 s/sec=2291.45 tok/sec=106619.36 u/sec=24.61 ppl=7.030278 [2023-12-17:11:50:36:INFO:sockeye.training:__call__] E=10 B=41300 s/sec=2286.89 tok/sec=103225.38 u/sec=24.94 ppl=7.043053 [2023-12-17:11:50:38:INFO:sockeye.training:__call__] E=10 B=41350 s/sec=2377.58 tok/sec=104187.02 u/sec=25.62 ppl=7.053045 [2023-12-17:11:50:40:INFO:sockeye.training:__call__] E=10 B=41400 s/sec=2522.30 tok/sec=102463.79 u/sec=25.06 ppl=7.069603 [2023-12-17:11:50:42:INFO:sockeye.training:__call__] E=10 B=41450 s/sec=2250.81 tok/sec=105100.97 u/sec=24.68 ppl=7.081377 [2023-12-17:11:50:44:INFO:sockeye.training:__call__] E=10 B=41500 s/sec=2096.92 tok/sec=108849.30 u/sec=24.82 ppl=7.086101 [2023-12-17:11:50:46:INFO:sockeye.training:__call__] E=10 B=41550 s/sec=2347.85 tok/sec=105689.28 u/sec=25.00 ppl=7.097213 [2023-12-17:11:50:48:INFO:sockeye.training:__call__] E=10 B=41600 s/sec=1990.49 tok/sec=108998.03 u/sec=24.73 ppl=7.103843 [2023-12-17:11:50:50:INFO:sockeye.training:__call__] E=10 B=41650 s/sec=2029.12 tok/sec=107257.81 u/sec=24.77 ppl=7.112308 [2023-12-17:11:50:52:INFO:sockeye.training:__call__] E=10 B=41700 s/sec=2185.21 tok/sec=108330.19 u/sec=24.43 ppl=7.125304 [2023-12-17:11:50:54:INFO:sockeye.training:__call__] E=10 B=41750 s/sec=2064.81 tok/sec=108702.50 u/sec=24.49 ppl=7.134897 [2023-12-17:11:50:56:INFO:sockeye.training:__call__] E=10 B=41800 s/sec=2087.79 tok/sec=108421.71 u/sec=24.85 ppl=7.143447 [2023-12-17:11:50:58:INFO:sockeye.training:__call__] E=10 B=41850 s/sec=2206.53 tok/sec=106200.68 u/sec=24.94 ppl=7.152253 [2023-12-17:11:51:00:INFO:sockeye.training:__call__] E=10 B=41900 s/sec=2265.03 tok/sec=107507.26 u/sec=24.97 ppl=7.165264 [2023-12-17:11:51:02:INFO:sockeye.training:__call__] E=10 B=41950 s/sec=2295.68 tok/sec=108387.88 u/sec=24.74 ppl=7.175443 [2023-12-17:11:51:04:INFO:sockeye.training:__call__] E=10 B=42000 s/sec=2223.96 tok/sec=105942.93 u/sec=25.23 ppl=7.183586 [2023-12-17:11:51:06:INFO:sockeye.training:__call__] E=10 B=42050 s/sec=2148.39 tok/sec=106359.71 u/sec=25.29 ppl=7.192649 [2023-12-17:11:51:08:INFO:sockeye.training:__call__] E=10 B=42100 s/sec=2075.54 tok/sec=107817.39 u/sec=24.71 ppl=7.200606 [2023-12-17:11:51:10:INFO:sockeye.training:__call__] E=10 B=42150 s/sec=2206.48 tok/sec=110051.35 u/sec=24.32 ppl=7.209103 [2023-12-17:11:51:12:INFO:sockeye.training:__call__] E=10 B=42200 s/sec=2106.05 tok/sec=108516.43 u/sec=24.38 ppl=7.216821 [2023-12-17:11:51:14:INFO:sockeye.training:__call__] E=10 B=42250 s/sec=2131.02 tok/sec=106606.10 u/sec=24.62 ppl=7.224222 [2023-12-17:11:51:16:INFO:sockeye.training:__call__] E=10 B=42300 s/sec=2171.41 tok/sec=107394.89 u/sec=24.99 ppl=7.228435 [2023-12-17:11:51:18:INFO:sockeye.training:__call__] E=10 B=42350 s/sec=2239.72 tok/sec=105198.74 u/sec=25.04 ppl=7.237987 [2023-12-17:11:51:20:INFO:sockeye.training:__call__] E=10 B=42400 s/sec=2406.51 tok/sec=105210.28 u/sec=25.15 ppl=7.250051 [2023-12-17:11:51:22:INFO:sockeye.training:__call__] E=10 B=42450 s/sec=2077.85 tok/sec=107877.87 u/sec=24.18 ppl=7.256040 [2023-12-17:11:51:24:INFO:sockeye.training:__call__] E=10 B=42500 s/sec=2129.55 tok/sec=106617.79 u/sec=24.38 ppl=7.263198 [2023-12-17:11:51:26:INFO:sockeye.training:__call__] E=10 B=42550 s/sec=2146.50 tok/sec=108787.72 u/sec=24.84 ppl=7.269466 [2023-12-17:11:51:28:INFO:sockeye.training:__call__] E=10 B=42600 s/sec=2077.69 tok/sec=108134.85 u/sec=24.78 ppl=7.275908 [2023-12-17:11:51:30:INFO:sockeye.training:__call__] E=10 B=42650 s/sec=2295.46 tok/sec=103141.77 u/sec=25.35 ppl=7.284028 [2023-12-17:11:51:32:INFO:sockeye.training:__call__] E=10 B=42700 s/sec=2085.96 tok/sec=110642.67 u/sec=24.60 ppl=7.289186 [2023-12-17:11:51:34:INFO:sockeye.training:__call__] E=10 B=42750 s/sec=2308.99 tok/sec=107763.29 u/sec=24.97 ppl=7.298048 [2023-12-17:11:51:36:INFO:sockeye.training:__call__] E=10 B=42800 s/sec=2489.68 tok/sec=103157.78 u/sec=25.59 ppl=7.309799 [2023-12-17:11:51:38:INFO:sockeye.training:__call__] E=10 B=42850 s/sec=2394.24 tok/sec=105672.38 u/sec=25.45 ppl=7.318133 [2023-12-17:11:51:40:INFO:sockeye.training:__call__] E=10 B=42900 s/sec=2139.10 tok/sec=107225.57 u/sec=24.18 ppl=7.328284 [2023-12-17:11:51:42:INFO:sockeye.training:__call__] E=10 B=42950 s/sec=2371.67 tok/sec=104288.86 u/sec=25.30 ppl=7.336873 [2023-12-17:11:51:44:INFO:sockeye.training:__call__] E=10 B=43000 s/sec=2525.32 tok/sec=100979.70 u/sec=25.83 ppl=7.347680 [2023-12-17:11:51:46:INFO:sockeye.training:__call__] E=10 B=43050 s/sec=2175.00 tok/sec=104790.43 u/sec=25.41 ppl=7.355013 [2023-12-17:11:51:48:INFO:sockeye.training:__call__] E=10 B=43100 s/sec=2401.35 tok/sec=104231.38 u/sec=25.35 ppl=7.361701 [2023-12-17:11:51:50:INFO:sockeye.training:__call__] E=10 B=43150 s/sec=2399.74 tok/sec=105140.73 u/sec=25.00 ppl=7.370382 [2023-12-17:11:51:52:INFO:sockeye.training:__call__] E=10 B=43200 s/sec=2053.14 tok/sec=114769.94 u/sec=22.96 ppl=7.378951 [2023-12-17:11:51:54:INFO:sockeye.training:__call__] E=10 B=43250 s/sec=1835.19 tok/sec=109634.96 u/sec=24.83 ppl=7.382230 [2023-12-17:11:51:56:INFO:sockeye.training:__call__] E=10 B=43300 s/sec=2203.80 tok/sec=105132.94 u/sec=25.37 ppl=7.387944 [2023-12-17:11:51:58:INFO:sockeye.training:__call__] E=10 B=43350 s/sec=2346.95 tok/sec=106011.77 u/sec=24.95 ppl=7.394608 [2023-12-17:11:52:00:INFO:sockeye.training:__call__] E=10 B=43400 s/sec=2363.16 tok/sec=104906.40 u/sec=25.42 ppl=7.404054 [2023-12-17:11:52:02:INFO:sockeye.training:__call__] E=10 B=43450 s/sec=2259.91 tok/sec=107751.32 u/sec=24.61 ppl=7.413185 [2023-12-17:11:52:04:INFO:sockeye.training:__call__] E=10 B=43500 s/sec=2266.06 tok/sec=105282.17 u/sec=25.47 ppl=7.419161 [2023-12-17:11:52:06:INFO:sockeye.training:__call__] E=10 B=43550 s/sec=2169.82 tok/sec=107622.86 u/sec=24.43 ppl=7.427068 [2023-12-17:11:52:08:INFO:sockeye.training:__call__] E=10 B=43600 s/sec=2204.44 tok/sec=107018.63 u/sec=24.78 ppl=7.434397 [2023-12-17:11:52:10:INFO:sockeye.training:__call__] E=10 B=43650 s/sec=2550.10 tok/sec=103173.60 u/sec=25.38 ppl=7.443040 [2023-12-17:11:52:12:INFO:sockeye.training:__call__] E=10 B=43700 s/sec=1874.46 tok/sec=110754.30 u/sec=24.56 ppl=7.447242 [2023-12-17:11:52:14:INFO:sockeye.training:__call__] E=10 B=43750 s/sec=2347.98 tok/sec=102165.88 u/sec=25.57 ppl=7.456787 [2023-12-17:11:52:16:INFO:sockeye.training:__call__] E=10 B=43800 s/sec=2297.82 tok/sec=105523.39 u/sec=25.02 ppl=7.462837 [2023-12-17:11:52:18:INFO:sockeye.training:__call__] E=10 B=43850 s/sec=1784.26 tok/sec=112369.75 u/sec=24.14 ppl=7.467063 [2023-12-17:11:52:20:INFO:sockeye.training:__call__] E=10 B=43900 s/sec=2145.70 tok/sec=108058.78 u/sec=24.43 ppl=7.474371 [2023-12-17:11:52:22:INFO:sockeye.training:__call__] E=11 B=43950 s/sec=2096.90 tok/sec=101106.15 u/sec=23.70 ppl=7.478648 [2023-12-17:11:52:24:INFO:sockeye.training:__call__] E=11 B=44000 s/sec=2210.24 tok/sec=105898.46 u/sec=25.12 ppl=7.466368 [2023-12-17:11:52:24:INFO:sockeye.training:_create_checkpoint] Checkpoint [11] Updates=44000 Epoch=11 Samples=3921272 Time-cost=160.749 Updates/sec=24.884 [2023-12-17:11:52:24:INFO:sockeye.training:_create_checkpoint] Checkpoint [11] Train-ppl=7.466368 [2023-12-17:11:52:27:INFO:sockeye.training:_evaluate] Checkpoint [11] Validation-ppl=31.358932 Validation-bleu=0.159950 Validation-chrf=30.898494 Validation-rouge1=0.165754 Validation-rouge2=0.077838 Validation-rougel=0.158876 Validation-length-ratio-mse=1.300733 Validation-ter=97.234833 Validation-avg-sec-per-sent=0.003345 Validation-decode-walltime=1.672364 [2023-12-17:11:52:27:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 6 checkpoints, best so far: 31.983236 [2023-12-17:11:52:27:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [11] Learning-rate=0.000200 [2023-12-17:11:52:27:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:52:27:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=6, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:52:28:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00011" [2023-12-17:11:52:30:INFO:sockeye.training:__call__] E=11 B=44050 s/sec=787.44 tok/sec=43343.75 u/sec=9.34 ppl=6.596199 [2023-12-17:11:52:32:INFO:sockeye.training:__call__] E=11 B=44100 s/sec=2244.31 tok/sec=105004.66 u/sec=25.32 ppl=6.575996 [2023-12-17:11:52:34:INFO:sockeye.training:__call__] E=11 B=44150 s/sec=2482.25 tok/sec=103177.29 u/sec=25.31 ppl=6.622721 [2023-12-17:11:52:36:INFO:sockeye.training:__call__] E=11 B=44200 s/sec=1995.54 tok/sec=110228.07 u/sec=24.26 ppl=6.602250 [2023-12-17:11:52:38:INFO:sockeye.training:__call__] E=11 B=44250 s/sec=2274.70 tok/sec=105184.54 u/sec=25.21 ppl=6.597226 [2023-12-17:11:52:40:INFO:sockeye.training:__call__] E=11 B=44300 s/sec=2328.13 tok/sec=105987.59 u/sec=24.87 ppl=6.620737 [2023-12-17:11:52:42:INFO:sockeye.training:__call__] E=11 B=44350 s/sec=2264.14 tok/sec=105971.20 u/sec=25.18 ppl=6.631284 [2023-12-17:11:52:44:INFO:sockeye.training:__call__] E=11 B=44400 s/sec=2237.56 tok/sec=105474.43 u/sec=25.11 ppl=6.637561 [2023-12-17:11:52:46:INFO:sockeye.training:__call__] E=11 B=44450 s/sec=2341.67 tok/sec=105975.43 u/sec=24.85 ppl=6.649298 [2023-12-17:11:52:48:INFO:sockeye.training:__call__] E=11 B=44500 s/sec=2103.79 tok/sec=107810.13 u/sec=25.05 ppl=6.659105 [2023-12-17:11:52:50:INFO:sockeye.training:__call__] E=11 B=44550 s/sec=2227.48 tok/sec=108362.00 u/sec=24.51 ppl=6.674955 [2023-12-17:11:52:52:INFO:sockeye.training:__call__] E=11 B=44600 s/sec=2153.35 tok/sec=106882.20 u/sec=24.79 ppl=6.696619 [2023-12-17:11:52:54:INFO:sockeye.training:__call__] E=11 B=44650 s/sec=1966.04 tok/sec=111325.06 u/sec=24.58 ppl=6.699105 [2023-12-17:11:52:56:INFO:sockeye.training:__call__] E=11 B=44700 s/sec=2215.79 tok/sec=107736.22 u/sec=25.04 ppl=6.706164 [2023-12-17:11:52:58:INFO:sockeye.training:__call__] E=11 B=44750 s/sec=2391.01 tok/sec=105739.45 u/sec=24.58 ppl=6.722795 [2023-12-17:11:53:00:INFO:sockeye.training:__call__] E=11 B=44800 s/sec=2008.53 tok/sec=108803.32 u/sec=24.33 ppl=6.725531 [2023-12-17:11:53:02:INFO:sockeye.training:__call__] E=11 B=44850 s/sec=2066.44 tok/sec=108563.39 u/sec=24.74 ppl=6.734768 [2023-12-17:11:53:04:INFO:sockeye.training:__call__] E=11 B=44900 s/sec=2554.94 tok/sec=100547.68 u/sec=26.31 ppl=6.752349 [2023-12-17:11:53:06:INFO:sockeye.training:__call__] E=11 B=44950 s/sec=2224.97 tok/sec=105788.72 u/sec=24.66 ppl=6.764498 [2023-12-17:11:53:08:INFO:sockeye.training:__call__] E=11 B=45000 s/sec=2179.91 tok/sec=103953.71 u/sec=25.37 ppl=6.777333 [2023-12-17:11:53:10:INFO:sockeye.training:__call__] E=11 B=45050 s/sec=2254.45 tok/sec=104212.95 u/sec=25.21 ppl=6.787883 [2023-12-17:11:53:12:INFO:sockeye.training:__call__] E=11 B=45100 s/sec=2236.32 tok/sec=106146.55 u/sec=24.61 ppl=6.802996 [2023-12-17:11:53:14:INFO:sockeye.training:__call__] E=11 B=45150 s/sec=2261.19 tok/sec=103597.45 u/sec=25.56 ppl=6.812867 [2023-12-17:11:53:16:INFO:sockeye.training:__call__] E=11 B=45200 s/sec=2177.90 tok/sec=108170.21 u/sec=24.88 ppl=6.821587 [2023-12-17:11:53:18:INFO:sockeye.training:__call__] E=11 B=45250 s/sec=1883.10 tok/sec=112955.16 u/sec=24.32 ppl=6.824660 [2023-12-17:11:53:20:INFO:sockeye.training:__call__] E=11 B=45300 s/sec=2269.35 tok/sec=104065.85 u/sec=25.33 ppl=6.838150 [2023-12-17:11:53:22:INFO:sockeye.training:__call__] E=11 B=45350 s/sec=2097.39 tok/sec=108536.86 u/sec=24.97 ppl=6.848868 [2023-12-17:11:53:24:INFO:sockeye.training:__call__] E=11 B=45400 s/sec=2018.93 tok/sec=105750.52 u/sec=24.94 ppl=6.856189 [2023-12-17:11:53:26:INFO:sockeye.training:__call__] E=11 B=45450 s/sec=2125.67 tok/sec=107700.44 u/sec=24.88 ppl=6.865323 [2023-12-17:11:53:28:INFO:sockeye.training:__call__] E=11 B=45500 s/sec=2398.02 tok/sec=103065.73 u/sec=25.49 ppl=6.876924 [2023-12-17:11:53:30:INFO:sockeye.training:__call__] E=11 B=45550 s/sec=2477.19 tok/sec=101696.77 u/sec=25.80 ppl=6.893636 [2023-12-17:11:53:32:INFO:sockeye.training:__call__] E=11 B=45600 s/sec=2317.13 tok/sec=104597.39 u/sec=25.50 ppl=6.907153 [2023-12-17:11:53:34:INFO:sockeye.training:__call__] E=11 B=45650 s/sec=2157.02 tok/sec=109715.49 u/sec=24.38 ppl=6.914262 [2023-12-17:11:53:36:INFO:sockeye.training:__call__] E=11 B=45700 s/sec=2292.51 tok/sec=106221.12 u/sec=24.66 ppl=6.923260 [2023-12-17:11:53:38:INFO:sockeye.training:__call__] E=11 B=45750 s/sec=2023.85 tok/sec=108859.11 u/sec=24.80 ppl=6.933837 [2023-12-17:11:53:40:INFO:sockeye.training:__call__] E=11 B=45800 s/sec=2280.61 tok/sec=110380.01 u/sec=24.53 ppl=6.939693 [2023-12-17:11:53:42:INFO:sockeye.training:__call__] E=11 B=45850 s/sec=2134.06 tok/sec=106814.12 u/sec=24.79 ppl=6.948225 [2023-12-17:11:53:44:INFO:sockeye.training:__call__] E=11 B=45900 s/sec=2168.13 tok/sec=106439.96 u/sec=24.68 ppl=6.956051 [2023-12-17:11:53:46:INFO:sockeye.training:__call__] E=11 B=45950 s/sec=2340.20 tok/sec=103789.86 u/sec=25.66 ppl=6.967077 [2023-12-17:11:53:48:INFO:sockeye.training:__call__] E=11 B=46000 s/sec=2261.65 tok/sec=102615.58 u/sec=25.65 ppl=6.974741 [2023-12-17:11:53:50:INFO:sockeye.training:__call__] E=11 B=46050 s/sec=2418.81 tok/sec=104551.23 u/sec=25.49 ppl=6.981922 [2023-12-17:11:53:52:INFO:sockeye.training:__call__] E=11 B=46100 s/sec=2423.33 tok/sec=103230.71 u/sec=25.76 ppl=6.992465 [2023-12-17:11:53:54:INFO:sockeye.training:__call__] E=11 B=46150 s/sec=1962.09 tok/sec=109156.55 u/sec=24.48 ppl=6.998839 [2023-12-17:11:53:56:INFO:sockeye.training:__call__] E=11 B=46200 s/sec=2388.62 tok/sec=101717.46 u/sec=25.65 ppl=7.008542 [2023-12-17:11:53:58:INFO:sockeye.training:__call__] E=11 B=46250 s/sec=2205.77 tok/sec=107568.39 u/sec=24.93 ppl=7.013438 [2023-12-17:11:54:00:INFO:sockeye.training:__call__] E=11 B=46300 s/sec=2250.29 tok/sec=106890.78 u/sec=25.07 ppl=7.022492 [2023-12-17:11:54:02:INFO:sockeye.training:__call__] E=11 B=46350 s/sec=2048.99 tok/sec=109916.57 u/sec=24.49 ppl=7.031183 [2023-12-17:11:54:04:INFO:sockeye.training:__call__] E=11 B=46400 s/sec=2062.41 tok/sec=109338.77 u/sec=24.05 ppl=7.035314 [2023-12-17:11:54:06:INFO:sockeye.training:__call__] E=11 B=46450 s/sec=2053.54 tok/sec=108670.07 u/sec=24.97 ppl=7.041917 [2023-12-17:11:54:08:INFO:sockeye.training:__call__] E=11 B=46500 s/sec=2098.45 tok/sec=106311.13 u/sec=24.65 ppl=7.046487 [2023-12-17:11:54:10:INFO:sockeye.training:__call__] E=11 B=46550 s/sec=2224.77 tok/sec=108412.48 u/sec=24.39 ppl=7.054280 [2023-12-17:11:54:12:INFO:sockeye.training:__call__] E=11 B=46600 s/sec=2501.82 tok/sec=105554.89 u/sec=25.34 ppl=7.065062 [2023-12-17:11:54:14:INFO:sockeye.training:__call__] E=11 B=46650 s/sec=2449.26 tok/sec=106099.55 u/sec=25.30 ppl=7.076026 [2023-12-17:11:54:16:INFO:sockeye.training:__call__] E=11 B=46700 s/sec=2331.24 tok/sec=104258.07 u/sec=25.30 ppl=7.084515 [2023-12-17:11:54:18:INFO:sockeye.training:__call__] E=11 B=46750 s/sec=2111.38 tok/sec=108109.12 u/sec=24.80 ppl=7.090173 [2023-12-17:11:54:20:INFO:sockeye.training:__call__] E=11 B=46800 s/sec=2203.37 tok/sec=106312.70 u/sec=25.31 ppl=7.097801 [2023-12-17:11:54:22:INFO:sockeye.training:__call__] E=11 B=46850 s/sec=2638.10 tok/sec=101177.82 u/sec=25.72 ppl=7.109153 [2023-12-17:11:54:24:INFO:sockeye.training:__call__] E=11 B=46900 s/sec=2293.44 tok/sec=109329.76 u/sec=24.59 ppl=7.115623 [2023-12-17:11:54:26:INFO:sockeye.training:__call__] E=11 B=46950 s/sec=2347.57 tok/sec=106275.80 u/sec=24.66 ppl=7.125034 [2023-12-17:11:54:28:INFO:sockeye.training:__call__] E=11 B=47000 s/sec=2272.76 tok/sec=111965.12 u/sec=23.87 ppl=7.135330 [2023-12-17:11:54:30:INFO:sockeye.training:__call__] E=11 B=47050 s/sec=2298.52 tok/sec=103425.18 u/sec=25.56 ppl=7.141774 [2023-12-17:11:54:32:INFO:sockeye.training:__call__] E=11 B=47100 s/sec=2126.03 tok/sec=110475.77 u/sec=24.38 ppl=7.149675 [2023-12-17:11:54:34:INFO:sockeye.training:__call__] E=11 B=47150 s/sec=2095.29 tok/sec=109297.74 u/sec=24.34 ppl=7.159687 [2023-12-17:11:54:36:INFO:sockeye.training:__call__] E=11 B=47200 s/sec=2135.34 tok/sec=106182.29 u/sec=25.04 ppl=7.164829 [2023-12-17:11:54:38:INFO:sockeye.training:__call__] E=11 B=47250 s/sec=2285.04 tok/sec=104828.76 u/sec=25.28 ppl=7.175232 [2023-12-17:11:54:40:INFO:sockeye.training:__call__] E=11 B=47300 s/sec=2115.07 tok/sec=108492.52 u/sec=24.53 ppl=7.180509 [2023-12-17:11:54:42:INFO:sockeye.training:__call__] E=11 B=47350 s/sec=2240.49 tok/sec=106771.24 u/sec=25.14 ppl=7.187401 [2023-12-17:11:54:44:INFO:sockeye.training:__call__] E=11 B=47400 s/sec=2127.58 tok/sec=107407.41 u/sec=24.72 ppl=7.192464 [2023-12-17:11:54:46:INFO:sockeye.training:__call__] E=11 B=47450 s/sec=2050.90 tok/sec=109244.79 u/sec=24.60 ppl=7.198408 [2023-12-17:11:54:48:INFO:sockeye.training:__call__] E=11 B=47500 s/sec=2036.55 tok/sec=109646.13 u/sec=24.38 ppl=7.205921 [2023-12-17:11:54:50:INFO:sockeye.training:__call__] E=11 B=47550 s/sec=2280.40 tok/sec=105940.80 u/sec=25.45 ppl=7.211672 [2023-12-17:11:54:52:INFO:sockeye.training:__call__] E=11 B=47600 s/sec=1968.45 tok/sec=111721.44 u/sec=24.22 ppl=7.215685 [2023-12-17:11:54:54:INFO:sockeye.training:__call__] E=11 B=47650 s/sec=2228.46 tok/sec=109413.11 u/sec=24.83 ppl=7.220136 [2023-12-17:11:54:56:INFO:sockeye.training:__call__] E=11 B=47700 s/sec=2296.61 tok/sec=104705.80 u/sec=25.27 ppl=7.226021 [2023-12-17:11:54:58:INFO:sockeye.training:__call__] E=11 B=47750 s/sec=2528.26 tok/sec=102856.51 u/sec=25.45 ppl=7.234587 [2023-12-17:11:55:00:INFO:sockeye.training:__call__] E=11 B=47800 s/sec=2336.39 tok/sec=106293.72 u/sec=24.58 ppl=7.241369 [2023-12-17:11:55:02:INFO:sockeye.training:__call__] E=11 B=47850 s/sec=2492.84 tok/sec=101947.65 u/sec=25.67 ppl=7.250633 [2023-12-17:11:55:04:INFO:sockeye.training:__call__] E=11 B=47900 s/sec=2173.41 tok/sec=108094.17 u/sec=24.34 ppl=7.258478 [2023-12-17:11:55:06:INFO:sockeye.training:__call__] E=12 B=47950 s/sec=2048.65 tok/sec=102652.72 u/sec=23.76 ppl=7.261173 [2023-12-17:11:55:08:INFO:sockeye.training:__call__] E=12 B=48000 s/sec=2044.64 tok/sec=109205.75 u/sec=24.77 ppl=7.249319 [2023-12-17:11:55:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [12] Updates=48000 Epoch=12 Samples=4277480 Time-cost=160.585 Updates/sec=24.909 [2023-12-17:11:55:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [12] Train-ppl=7.249319 [2023-12-17:11:55:29:INFO:sockeye.training:_evaluate] Checkpoint [12] Validation-ppl=33.066800 Validation-bleu=0.159157 Validation-chrf=29.846615 Validation-rouge1=0.145661 Validation-rouge2=0.071502 Validation-rougel=0.138529 Validation-length-ratio-mse=1.286198 Validation-ter=98.060256 Validation-avg-sec-per-sent=0.039493 Validation-decode-walltime=19.746372 [2023-12-17:11:55:29:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 7 checkpoints, best so far: 31.983236 [2023-12-17:11:55:29:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [12] Learning-rate=0.000200 [2023-12-17:11:55:29:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:55:29:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=7, base_lr=0.0002, lr=0.0002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:55:29:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00012" [2023-12-17:11:55:31:INFO:sockeye.training:__call__] E=12 B=48050 s/sec=187.20 tok/sec=8897.84 u/sec=2.15 ppl=6.426198 [2023-12-17:11:55:33:INFO:sockeye.training:__call__] E=12 B=48100 s/sec=2119.12 tok/sec=109438.02 u/sec=24.62 ppl=6.454798 [2023-12-17:11:55:35:INFO:sockeye.training:__call__] E=12 B=48150 s/sec=2248.92 tok/sec=108333.91 u/sec=25.10 ppl=6.435897 [2023-12-17:11:55:37:INFO:sockeye.training:__call__] E=12 B=48200 s/sec=2376.57 tok/sec=103574.25 u/sec=25.48 ppl=6.435910 [2023-12-17:11:55:39:INFO:sockeye.training:__call__] E=12 B=48250 s/sec=2078.96 tok/sec=107551.61 u/sec=24.75 ppl=6.436725 [2023-12-17:11:55:41:INFO:sockeye.training:__call__] E=12 B=48300 s/sec=2087.50 tok/sec=110539.43 u/sec=24.48 ppl=6.435905 [2023-12-17:11:55:43:INFO:sockeye.training:__call__] E=12 B=48350 s/sec=2362.76 tok/sec=104172.97 u/sec=25.42 ppl=6.456136 [2023-12-17:11:55:45:INFO:sockeye.training:__call__] E=12 B=48400 s/sec=2246.93 tok/sec=106600.66 u/sec=25.08 ppl=6.472729 [2023-12-17:11:55:48:INFO:sockeye.training:__call__] E=12 B=48450 s/sec=2013.69 tok/sec=110929.25 u/sec=23.93 ppl=6.469757 [2023-12-17:11:55:50:INFO:sockeye.training:__call__] E=12 B=48500 s/sec=2089.42 tok/sec=108571.04 u/sec=24.69 ppl=6.490189 [2023-12-17:11:55:52:INFO:sockeye.training:__call__] E=12 B=48550 s/sec=2256.98 tok/sec=106510.24 u/sec=25.01 ppl=6.511429 [2023-12-17:11:55:54:INFO:sockeye.training:__call__] E=12 B=48600 s/sec=2181.74 tok/sec=109158.15 u/sec=24.75 ppl=6.516178 [2023-12-17:11:55:56:INFO:sockeye.training:__call__] E=12 B=48650 s/sec=2245.82 tok/sec=104388.15 u/sec=25.38 ppl=6.528555 [2023-12-17:11:55:58:INFO:sockeye.training:__call__] E=12 B=48700 s/sec=2208.59 tok/sec=106592.44 u/sec=25.19 ppl=6.536831 [2023-12-17:11:55:59:INFO:sockeye.training:__call__] E=12 B=48750 s/sec=2413.68 tok/sec=103966.31 u/sec=25.88 ppl=6.558031 [2023-12-17:11:56:01:INFO:sockeye.training:__call__] E=12 B=48800 s/sec=2100.63 tok/sec=106718.43 u/sec=24.87 ppl=6.568260 [2023-12-17:11:56:03:INFO:sockeye.training:__call__] E=12 B=48850 s/sec=2173.56 tok/sec=106910.55 u/sec=24.88 ppl=6.580139 [2023-12-17:11:56:06:INFO:sockeye.training:__call__] E=12 B=48900 s/sec=2122.97 tok/sec=109939.42 u/sec=24.53 ppl=6.588230 [2023-12-17:11:56:08:INFO:sockeye.training:__call__] E=12 B=48950 s/sec=2006.20 tok/sec=108342.91 u/sec=24.63 ppl=6.598767 [2023-12-17:11:56:10:INFO:sockeye.training:__call__] E=12 B=49000 s/sec=2324.12 tok/sec=107575.63 u/sec=24.50 ppl=6.610129 [2023-12-17:11:56:12:INFO:sockeye.training:__call__] E=12 B=49050 s/sec=1990.24 tok/sec=107657.99 u/sec=25.13 ppl=6.613692 [2023-12-17:11:56:14:INFO:sockeye.training:__call__] E=12 B=49100 s/sec=2371.55 tok/sec=104187.61 u/sec=25.12 ppl=6.629302 [2023-12-17:11:56:16:INFO:sockeye.training:__call__] E=12 B=49150 s/sec=1876.46 tok/sec=114407.82 u/sec=23.50 ppl=6.630315 [2023-12-17:11:56:18:INFO:sockeye.training:__call__] E=12 B=49200 s/sec=2240.96 tok/sec=111963.77 u/sec=23.86 ppl=6.641323 [2023-12-17:11:56:20:INFO:sockeye.training:__call__] E=12 B=49250 s/sec=2056.58 tok/sec=108752.81 u/sec=24.81 ppl=6.647507 [2023-12-17:11:56:22:INFO:sockeye.training:__call__] E=12 B=49300 s/sec=2198.64 tok/sec=107721.15 u/sec=25.12 ppl=6.659405 [2023-12-17:11:56:24:INFO:sockeye.training:__call__] E=12 B=49350 s/sec=2177.21 tok/sec=107254.77 u/sec=24.97 ppl=6.666703 [2023-12-17:11:56:26:INFO:sockeye.training:__call__] E=12 B=49400 s/sec=2126.93 tok/sec=109091.75 u/sec=24.30 ppl=6.674925 [2023-12-17:11:56:28:INFO:sockeye.training:__call__] E=12 B=49450 s/sec=2281.88 tok/sec=106167.66 u/sec=25.02 ppl=6.683646 [2023-12-17:11:56:30:INFO:sockeye.training:__call__] E=12 B=49500 s/sec=2239.57 tok/sec=103967.64 u/sec=25.08 ppl=6.693209 [2023-12-17:11:56:32:INFO:sockeye.training:__call__] E=12 B=49550 s/sec=2120.48 tok/sec=106875.43 u/sec=25.34 ppl=6.702425 [2023-12-17:11:56:34:INFO:sockeye.training:__call__] E=12 B=49600 s/sec=2422.04 tok/sec=102760.38 u/sec=25.66 ppl=6.714384 [2023-12-17:11:56:36:INFO:sockeye.training:__call__] E=12 B=49650 s/sec=2546.20 tok/sec=104624.18 u/sec=25.22 ppl=6.726254 [2023-12-17:11:56:38:INFO:sockeye.training:__call__] E=12 B=49700 s/sec=2267.87 tok/sec=104840.60 u/sec=25.31 ppl=6.733634 [2023-12-17:11:56:40:INFO:sockeye.training:__call__] E=12 B=49750 s/sec=2392.56 tok/sec=104025.67 u/sec=25.30 ppl=6.749544 [2023-12-17:11:56:42:INFO:sockeye.training:__call__] E=12 B=49800 s/sec=2292.35 tok/sec=104965.08 u/sec=24.74 ppl=6.760886 [2023-12-17:11:56:44:INFO:sockeye.training:__call__] E=12 B=49850 s/sec=2399.28 tok/sec=105552.42 u/sec=25.12 ppl=6.770585 [2023-12-17:11:56:46:INFO:sockeye.training:__call__] E=12 B=49900 s/sec=2299.22 tok/sec=105364.12 u/sec=25.52 ppl=6.780839 [2023-12-17:11:56:48:INFO:sockeye.training:__call__] E=12 B=49950 s/sec=2401.64 tok/sec=104036.64 u/sec=24.81 ppl=6.791709 [2023-12-17:11:56:50:INFO:sockeye.training:__call__] E=12 B=50000 s/sec=1988.48 tok/sec=108376.01 u/sec=24.76 ppl=6.796775 [2023-12-17:11:56:52:INFO:sockeye.training:__call__] E=12 B=50050 s/sec=2621.11 tok/sec=98881.70 u/sec=25.88 ppl=6.809985 [2023-12-17:11:56:54:INFO:sockeye.training:__call__] E=12 B=50100 s/sec=2421.09 tok/sec=101308.20 u/sec=26.22 ppl=6.821791 [2023-12-17:11:56:56:INFO:sockeye.training:__call__] E=12 B=50150 s/sec=2310.29 tok/sec=106122.33 u/sec=24.85 ppl=6.825325 [2023-12-17:11:56:58:INFO:sockeye.training:__call__] E=12 B=50200 s/sec=2055.00 tok/sec=107700.75 u/sec=24.33 ppl=6.834425 [2023-12-17:11:57:00:INFO:sockeye.training:__call__] E=12 B=50250 s/sec=2288.31 tok/sec=107146.85 u/sec=24.74 ppl=6.842064 [2023-12-17:11:57:02:INFO:sockeye.training:__call__] E=12 B=50300 s/sec=2219.24 tok/sec=106555.35 u/sec=24.99 ppl=6.850033 [2023-12-17:11:57:04:INFO:sockeye.training:__call__] E=12 B=50350 s/sec=2109.10 tok/sec=107637.88 u/sec=24.28 ppl=6.859065 [2023-12-17:11:57:06:INFO:sockeye.training:__call__] E=12 B=50400 s/sec=2279.10 tok/sec=105081.90 u/sec=25.35 ppl=6.866874 [2023-12-17:11:57:08:INFO:sockeye.training:__call__] E=12 B=50450 s/sec=2149.34 tok/sec=106376.04 u/sec=25.06 ppl=6.873471 [2023-12-17:11:57:10:INFO:sockeye.training:__call__] E=12 B=50500 s/sec=2181.52 tok/sec=105872.28 u/sec=25.16 ppl=6.881956 [2023-12-17:11:57:12:INFO:sockeye.training:__call__] E=12 B=50550 s/sec=2513.93 tok/sec=104119.64 u/sec=25.30 ppl=6.892062 [2023-12-17:11:57:14:INFO:sockeye.training:__call__] E=12 B=50600 s/sec=1934.95 tok/sec=111409.45 u/sec=24.33 ppl=6.898520 [2023-12-17:11:57:16:INFO:sockeye.training:__call__] E=12 B=50650 s/sec=2164.21 tok/sec=108575.86 u/sec=24.28 ppl=6.905523 [2023-12-17:11:57:18:INFO:sockeye.training:__call__] E=12 B=50700 s/sec=2359.04 tok/sec=103233.20 u/sec=25.20 ppl=6.915770 [2023-12-17:11:57:20:INFO:sockeye.training:__call__] E=12 B=50750 s/sec=2077.66 tok/sec=109428.38 u/sec=24.14 ppl=6.921607 [2023-12-17:11:57:22:INFO:sockeye.training:__call__] E=12 B=50800 s/sec=2346.59 tok/sec=102743.76 u/sec=25.51 ppl=6.927190 [2023-12-17:11:57:24:INFO:sockeye.training:__call__] E=12 B=50850 s/sec=2349.12 tok/sec=103526.91 u/sec=25.85 ppl=6.933314 [2023-12-17:11:57:26:INFO:sockeye.training:__call__] E=12 B=50900 s/sec=2069.88 tok/sec=108287.78 u/sec=24.93 ppl=6.939084 [2023-12-17:11:57:28:INFO:sockeye.training:__call__] E=12 B=50950 s/sec=2419.61 tok/sec=103978.54 u/sec=25.20 ppl=6.946972 [2023-12-17:11:57:30:INFO:sockeye.training:__call__] E=12 B=51000 s/sec=2120.73 tok/sec=108486.27 u/sec=24.77 ppl=6.952618 [2023-12-17:11:57:32:INFO:sockeye.training:__call__] E=12 B=51050 s/sec=1998.05 tok/sec=108580.43 u/sec=24.63 ppl=6.960038 [2023-12-17:11:57:34:INFO:sockeye.training:__call__] E=12 B=51100 s/sec=2125.08 tok/sec=109639.81 u/sec=24.55 ppl=6.966718 [2023-12-17:11:57:36:INFO:sockeye.training:__call__] E=12 B=51150 s/sec=2272.70 tok/sec=106733.39 u/sec=24.88 ppl=6.974155 [2023-12-17:11:57:38:INFO:sockeye.training:__call__] E=12 B=51200 s/sec=2111.77 tok/sec=106606.79 u/sec=24.67 ppl=6.980222 [2023-12-17:11:57:40:INFO:sockeye.training:__call__] E=12 B=51250 s/sec=1914.46 tok/sec=110758.41 u/sec=24.32 ppl=6.984163 [2023-12-17:11:57:42:INFO:sockeye.training:__call__] E=12 B=51300 s/sec=2303.86 tok/sec=106574.55 u/sec=24.87 ppl=6.990335 [2023-12-17:11:57:44:INFO:sockeye.training:__call__] E=12 B=51350 s/sec=2401.88 tok/sec=104651.01 u/sec=25.19 ppl=6.997571 [2023-12-17:11:57:46:INFO:sockeye.training:__call__] E=12 B=51400 s/sec=2174.28 tok/sec=105967.33 u/sec=25.03 ppl=7.004640 [2023-12-17:11:57:48:INFO:sockeye.training:__call__] E=12 B=51450 s/sec=2049.84 tok/sec=106265.04 u/sec=25.52 ppl=7.010402 [2023-12-17:11:57:50:INFO:sockeye.training:__call__] E=12 B=51500 s/sec=2331.24 tok/sec=104023.06 u/sec=25.43 ppl=7.017109 [2023-12-17:11:57:52:INFO:sockeye.training:__call__] E=12 B=51550 s/sec=2391.87 tok/sec=106308.78 u/sec=24.87 ppl=7.024825 [2023-12-17:11:57:54:INFO:sockeye.training:__call__] E=12 B=51600 s/sec=2308.16 tok/sec=106087.42 u/sec=25.00 ppl=7.030282 [2023-12-17:11:57:56:INFO:sockeye.training:__call__] E=12 B=51650 s/sec=2270.90 tok/sec=108177.20 u/sec=24.81 ppl=7.037062 [2023-12-17:11:57:58:INFO:sockeye.training:__call__] E=12 B=51700 s/sec=2356.27 tok/sec=105663.43 u/sec=25.05 ppl=7.044281 [2023-12-17:11:58:00:INFO:sockeye.training:__call__] E=12 B=51750 s/sec=2296.79 tok/sec=105723.80 u/sec=24.88 ppl=7.049364 [2023-12-17:11:58:02:INFO:sockeye.training:__call__] E=12 B=51800 s/sec=2419.38 tok/sec=104952.98 u/sec=24.67 ppl=7.056394 [2023-12-17:11:58:04:INFO:sockeye.training:__call__] E=12 B=51850 s/sec=2249.81 tok/sec=107832.29 u/sec=24.80 ppl=7.062193 [2023-12-17:11:58:06:INFO:sockeye.training:__call__] E=12 B=51900 s/sec=2357.93 tok/sec=103157.19 u/sec=25.67 ppl=7.068610 [2023-12-17:11:58:08:INFO:sockeye.training:__call__] E=13 B=51950 s/sec=2025.37 tok/sec=104402.15 u/sec=23.49 ppl=7.069562 [2023-12-17:11:58:10:INFO:sockeye.training:__call__] E=13 B=52000 s/sec=2254.88 tok/sec=105130.78 u/sec=25.26 ppl=7.058896 [2023-12-17:11:58:10:INFO:sockeye.training:_create_checkpoint] Checkpoint [13] Updates=52000 Epoch=13 Samples=4634504 Time-cost=160.477 Updates/sec=24.926 [2023-12-17:11:58:10:INFO:sockeye.training:_create_checkpoint] Checkpoint [13] Train-ppl=7.058896 [2023-12-17:11:58:13:INFO:sockeye.training:_evaluate] Checkpoint [13] Validation-ppl=33.000284 Validation-bleu=0.155498 Validation-chrf=30.232177 Validation-rouge1=0.158555 Validation-rouge2=0.079668 Validation-rougel=0.151938 Validation-length-ratio-mse=1.332198 Validation-ter=99.793644 Validation-avg-sec-per-sent=0.003355 Validation-decode-walltime=1.677501 [2023-12-17:11:58:13:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 8 checkpoints, best so far: 31.983236 [2023-12-17:11:58:13:INFO:sockeye.lr_scheduler:new_evaluation_result] 8 checkpoints since improvement or rate scaling, lowering learning rate: 2.00e-04 -> 1.80e-04 [2023-12-17:11:58:13:INFO:sockeye.training:_adjust_learning_rate] Loading model parameters and optimizer states from best checkpoint: 5 [2023-12-17:11:58:13:INFO:sockeye.model:load_parameters] Loaded params from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best" to "cuda:0" [2023-12-17:11:58:13:INFO:sockeye.training:_load_optimizer_state] Loaded optimizer state from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl" [2023-12-17:11:58:13:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [13] Learning-rate=0.000200 [2023-12-17:11:58:13:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:11:58:13:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:11:58:14:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00013" [2023-12-17:11:58:16:INFO:sockeye.training:__call__] E=13 B=52050 s/sec=726.89 tok/sec=38007.32 u/sec=8.60 ppl=8.404862 [2023-12-17:11:58:18:INFO:sockeye.training:__call__] E=13 B=52100 s/sec=2008.05 tok/sec=99552.15 u/sec=23.11 ppl=8.350213 [2023-12-17:11:58:20:INFO:sockeye.training:__call__] E=13 B=52150 s/sec=2345.45 tok/sec=96481.33 u/sec=23.64 ppl=8.401378 [2023-12-17:11:58:22:INFO:sockeye.training:__call__] E=13 B=52200 s/sec=2040.64 tok/sec=101575.49 u/sec=22.30 ppl=8.393537 [2023-12-17:11:58:24:INFO:sockeye.training:__call__] E=13 B=52250 s/sec=2066.95 tok/sec=102392.57 u/sec=22.78 ppl=8.448532 [2023-12-17:11:58:27:INFO:sockeye.training:__call__] E=13 B=52300 s/sec=2065.01 tok/sec=97847.48 u/sec=23.13 ppl=8.445333 [2023-12-17:11:58:29:INFO:sockeye.training:__call__] E=13 B=52350 s/sec=2027.90 tok/sec=97915.24 u/sec=23.69 ppl=8.430912 [2023-12-17:11:58:31:INFO:sockeye.training:__call__] E=13 B=52400 s/sec=1934.62 tok/sec=100066.58 u/sec=23.16 ppl=8.433464 [2023-12-17:11:58:33:INFO:sockeye.training:__call__] E=13 B=52450 s/sec=2222.58 tok/sec=95385.25 u/sec=23.66 ppl=8.460779 [2023-12-17:11:58:35:INFO:sockeye.training:__call__] E=13 B=52500 s/sec=2141.58 tok/sec=97443.59 u/sec=23.36 ppl=8.473897 [2023-12-17:11:58:37:INFO:sockeye.training:__call__] E=13 B=52550 s/sec=1905.14 tok/sec=99957.05 u/sec=23.17 ppl=8.460962 [2023-12-17:11:58:40:INFO:sockeye.training:__call__] E=13 B=52600 s/sec=1917.57 tok/sec=101580.08 u/sec=22.78 ppl=8.439509 [2023-12-17:11:58:42:INFO:sockeye.training:__call__] E=13 B=52650 s/sec=2063.07 tok/sec=101787.77 u/sec=22.70 ppl=8.452924 [2023-12-17:11:58:44:INFO:sockeye.training:__call__] E=13 B=52700 s/sec=1895.45 tok/sec=104412.35 u/sec=22.56 ppl=8.457822 [2023-12-17:11:58:46:INFO:sockeye.training:__call__] E=13 B=52750 s/sec=2283.93 tok/sec=95254.98 u/sec=23.79 ppl=8.478810 [2023-12-17:11:58:48:INFO:sockeye.training:__call__] E=13 B=52800 s/sec=1881.25 tok/sec=98997.56 u/sec=23.19 ppl=8.470455 [2023-12-17:11:58:50:INFO:sockeye.training:__call__] E=13 B=52850 s/sec=1974.73 tok/sec=100359.12 u/sec=23.16 ppl=8.467060 [2023-12-17:11:58:52:INFO:sockeye.training:__call__] E=13 B=52900 s/sec=2433.91 tok/sec=91240.33 u/sec=24.38 ppl=8.493150 [2023-12-17:11:58:55:INFO:sockeye.training:__call__] E=13 B=52950 s/sec=2214.29 tok/sec=95566.65 u/sec=23.66 ppl=8.505083 [2023-12-17:11:58:57:INFO:sockeye.training:__call__] E=13 B=53000 s/sec=2004.42 tok/sec=109002.22 u/sec=21.31 ppl=8.503180 [2023-12-17:11:58:59:INFO:sockeye.training:__call__] E=13 B=53050 s/sec=2093.70 tok/sec=98793.47 u/sec=22.96 ppl=8.521538 [2023-12-17:11:59:01:INFO:sockeye.training:__call__] E=13 B=53100 s/sec=1819.17 tok/sec=100948.78 u/sec=23.30 ppl=8.526166 [2023-12-17:11:59:03:INFO:sockeye.training:__call__] E=13 B=53150 s/sec=2001.61 tok/sec=101171.15 u/sec=22.87 ppl=8.535924 [2023-12-17:11:59:06:INFO:sockeye.training:__call__] E=13 B=53200 s/sec=2148.46 tok/sec=98223.99 u/sec=23.35 ppl=8.559269 [2023-12-17:11:59:08:INFO:sockeye.training:__call__] E=13 B=53250 s/sec=1965.06 tok/sec=100311.68 u/sec=23.39 ppl=8.567698 [2023-12-17:11:59:10:INFO:sockeye.training:__call__] E=13 B=53300 s/sec=1675.42 tok/sec=106716.59 u/sec=22.14 ppl=8.565260 [2023-12-17:11:59:12:INFO:sockeye.training:__call__] E=13 B=53350 s/sec=1850.23 tok/sec=102123.77 u/sec=22.81 ppl=8.576345 [2023-12-17:11:59:14:INFO:sockeye.training:__call__] E=13 B=53400 s/sec=2271.97 tok/sec=97415.05 u/sec=23.32 ppl=8.591455 [2023-12-17:11:59:17:INFO:sockeye.training:__call__] E=13 B=53450 s/sec=2118.48 tok/sec=98152.82 u/sec=22.63 ppl=8.609430 [2023-12-17:11:59:19:INFO:sockeye.training:__call__] E=13 B=53500 s/sec=2211.54 tok/sec=94936.43 u/sec=23.75 ppl=8.630853 [2023-12-17:11:59:21:INFO:sockeye.training:__call__] E=13 B=53550 s/sec=1908.99 tok/sec=101900.32 u/sec=22.73 ppl=8.638923 [2023-12-17:11:59:23:INFO:sockeye.training:__call__] E=13 B=53600 s/sec=2108.03 tok/sec=96141.52 u/sec=23.74 ppl=8.657415 [2023-12-17:11:59:25:INFO:sockeye.training:__call__] E=13 B=53650 s/sec=2017.01 tok/sec=98326.53 u/sec=23.13 ppl=8.660521 [2023-12-17:11:59:27:INFO:sockeye.training:__call__] E=13 B=53700 s/sec=2213.12 tok/sec=96306.73 u/sec=23.56 ppl=8.677801 [2023-12-17:11:59:29:INFO:sockeye.training:__call__] E=13 B=53750 s/sec=2173.43 tok/sec=99008.37 u/sec=22.95 ppl=8.689920 [2023-12-17:11:59:32:INFO:sockeye.training:__call__] E=13 B=53800 s/sec=1997.61 tok/sec=99954.40 u/sec=23.04 ppl=8.696036 [2023-12-17:11:59:34:INFO:sockeye.training:__call__] E=13 B=53850 s/sec=2247.93 tok/sec=94307.66 u/sec=23.49 ppl=8.711733 [2023-12-17:11:59:36:INFO:sockeye.training:__call__] E=13 B=53900 s/sec=2182.22 tok/sec=96756.21 u/sec=23.56 ppl=8.724537 [2023-12-17:11:59:38:INFO:sockeye.training:__call__] E=13 B=53950 s/sec=1927.66 tok/sec=101418.82 u/sec=22.77 ppl=8.733685 [2023-12-17:11:59:40:INFO:sockeye.training:__call__] E=13 B=54000 s/sec=2036.95 tok/sec=98726.71 u/sec=23.27 ppl=8.747541 [2023-12-17:11:59:42:INFO:sockeye.training:__call__] E=13 B=54050 s/sec=2096.13 tok/sec=100147.79 u/sec=22.78 ppl=8.757429 [2023-12-17:11:59:45:INFO:sockeye.training:__call__] E=13 B=54100 s/sec=1969.68 tok/sec=103610.15 u/sec=22.34 ppl=8.768757 [2023-12-17:11:59:47:INFO:sockeye.training:__call__] E=13 B=54150 s/sec=2228.93 tok/sec=96634.83 u/sec=23.65 ppl=8.783008 [2023-12-17:11:59:49:INFO:sockeye.training:__call__] E=13 B=54200 s/sec=2023.79 tok/sec=99109.71 u/sec=23.29 ppl=8.791716 [2023-12-17:11:59:51:INFO:sockeye.training:__call__] E=13 B=54250 s/sec=2010.79 tok/sec=100232.67 u/sec=23.40 ppl=8.800845 [2023-12-17:11:59:53:INFO:sockeye.training:__call__] E=13 B=54300 s/sec=2295.99 tok/sec=94875.44 u/sec=23.96 ppl=8.815299 [2023-12-17:11:59:55:INFO:sockeye.training:__call__] E=13 B=54350 s/sec=2077.72 tok/sec=98015.10 u/sec=23.11 ppl=8.826372 [2023-12-17:11:59:57:INFO:sockeye.training:__call__] E=13 B=54400 s/sec=1836.89 tok/sec=102077.24 u/sec=22.82 ppl=8.831585 [2023-12-17:12:00:00:INFO:sockeye.training:__call__] E=13 B=54450 s/sec=2044.76 tok/sec=97728.83 u/sec=23.41 ppl=8.842119 [2023-12-17:12:00:02:INFO:sockeye.training:__call__] E=13 B=54500 s/sec=1965.84 tok/sec=100073.58 u/sec=23.01 ppl=8.848059 [2023-12-17:12:00:04:INFO:sockeye.training:__call__] E=13 B=54550 s/sec=2025.15 tok/sec=99877.29 u/sec=23.44 ppl=8.859884 [2023-12-17:12:00:06:INFO:sockeye.training:__call__] E=13 B=54600 s/sec=2328.71 tok/sec=95087.07 u/sec=23.67 ppl=8.870886 [2023-12-17:12:00:08:INFO:sockeye.training:__call__] E=13 B=54650 s/sec=2207.39 tok/sec=99123.65 u/sec=23.30 ppl=8.883427 [2023-12-17:12:00:10:INFO:sockeye.training:__call__] E=13 B=54700 s/sec=2165.66 tok/sec=96187.44 u/sec=23.79 ppl=8.891843 [2023-12-17:12:00:12:INFO:sockeye.training:__call__] E=13 B=54750 s/sec=1602.75 tok/sec=102719.44 u/sec=22.46 ppl=8.892550 [2023-12-17:12:00:15:INFO:sockeye.training:__call__] E=13 B=54800 s/sec=2039.83 tok/sec=98763.02 u/sec=22.93 ppl=8.901786 [2023-12-17:12:00:17:INFO:sockeye.training:__call__] E=13 B=54850 s/sec=2270.48 tok/sec=96177.55 u/sec=23.65 ppl=8.912370 [2023-12-17:12:00:19:INFO:sockeye.training:__call__] E=13 B=54900 s/sec=2087.58 tok/sec=98315.00 u/sec=23.42 ppl=8.918003 [2023-12-17:12:00:21:INFO:sockeye.training:__call__] E=13 B=54950 s/sec=2118.49 tok/sec=98823.49 u/sec=23.07 ppl=8.923288 [2023-12-17:12:00:23:INFO:sockeye.training:__call__] E=13 B=55000 s/sec=1904.59 tok/sec=102760.10 u/sec=22.89 ppl=8.928414 [2023-12-17:12:00:25:INFO:sockeye.training:__call__] E=13 B=55050 s/sec=2100.08 tok/sec=95478.92 u/sec=23.91 ppl=8.936910 [2023-12-17:12:00:28:INFO:sockeye.training:__call__] E=13 B=55100 s/sec=1865.74 tok/sec=102543.60 u/sec=22.60 ppl=8.942789 [2023-12-17:12:00:30:INFO:sockeye.training:__call__] E=13 B=55150 s/sec=2114.24 tok/sec=98815.87 u/sec=22.90 ppl=8.954341 [2023-12-17:12:00:32:INFO:sockeye.training:__call__] E=13 B=55200 s/sec=1858.20 tok/sec=100521.13 u/sec=23.23 ppl=8.961992 [2023-12-17:12:00:34:INFO:sockeye.training:__call__] E=13 B=55250 s/sec=2245.44 tok/sec=98073.37 u/sec=23.63 ppl=8.973448 [2023-12-17:12:00:36:INFO:sockeye.training:__call__] E=13 B=55300 s/sec=2046.55 tok/sec=100047.12 u/sec=23.21 ppl=8.982648 [2023-12-17:12:00:38:INFO:sockeye.training:__call__] E=13 B=55350 s/sec=2209.19 tok/sec=95612.75 u/sec=23.68 ppl=8.991393 [2023-12-17:12:00:40:INFO:sockeye.training:__call__] E=13 B=55400 s/sec=2197.80 tok/sec=97941.97 u/sec=23.32 ppl=8.999314 [2023-12-17:12:00:43:INFO:sockeye.training:__call__] E=13 B=55450 s/sec=2273.91 tok/sec=97453.41 u/sec=23.61 ppl=9.010148 [2023-12-17:12:00:45:INFO:sockeye.training:__call__] E=13 B=55500 s/sec=1891.25 tok/sec=102819.95 u/sec=22.78 ppl=9.016816 [2023-12-17:12:00:47:INFO:sockeye.training:__call__] E=13 B=55550 s/sec=2037.20 tok/sec=99527.45 u/sec=23.07 ppl=9.023608 [2023-12-17:12:00:49:INFO:sockeye.training:__call__] E=13 B=55600 s/sec=1930.95 tok/sec=101424.63 u/sec=22.99 ppl=9.030694 [2023-12-17:12:00:51:INFO:sockeye.training:__call__] E=13 B=55650 s/sec=2211.75 tok/sec=97779.49 u/sec=23.31 ppl=9.039072 [2023-12-17:12:00:53:INFO:sockeye.training:__call__] E=13 B=55700 s/sec=2171.35 tok/sec=96711.62 u/sec=23.48 ppl=9.046591 [2023-12-17:12:00:56:INFO:sockeye.training:__call__] E=13 B=55750 s/sec=2032.78 tok/sec=100825.69 u/sec=23.10 ppl=9.052814 [2023-12-17:12:00:58:INFO:sockeye.training:__call__] E=13 B=55800 s/sec=2104.47 tok/sec=102400.45 u/sec=22.56 ppl=9.061085 [2023-12-17:12:01:00:INFO:sockeye.training:__call__] E=13 B=55850 s/sec=2232.20 tok/sec=96052.05 u/sec=23.49 ppl=9.069080 [2023-12-17:12:01:02:INFO:sockeye.training:__call__] E=13 B=55900 s/sec=2007.49 tok/sec=97896.36 u/sec=23.54 ppl=9.075221 [2023-12-17:12:01:04:INFO:sockeye.training:__call__] E=14 B=55950 s/sec=2035.50 tok/sec=95284.25 u/sec=22.24 ppl=9.072330 [2023-12-17:12:01:06:INFO:sockeye.training:__call__] E=14 B=56000 s/sec=2063.46 tok/sec=99808.38 u/sec=22.91 ppl=9.054902 [2023-12-17:12:01:06:INFO:sockeye.training:_create_checkpoint] Checkpoint [14] Updates=56000 Epoch=14 Samples=4991032 Time-cost=172.854 Updates/sec=23.141 [2023-12-17:12:01:06:INFO:sockeye.training:_create_checkpoint] Checkpoint [14] Train-ppl=9.054902 [2023-12-17:12:01:09:INFO:sockeye.training:_evaluate] Checkpoint [14] Validation-ppl=27.111713 Validation-bleu=0.144674 Validation-chrf=30.229130 Validation-rouge1=0.157525 Validation-rouge2=0.073042 Validation-rougel=0.152280 Validation-length-ratio-mse=1.289199 Validation-ter=96.491952 Validation-avg-sec-per-sent=0.003481 Validation-decode-walltime=1.740565 [2023-12-17:12:01:09:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 9 checkpoints, best so far: 31.983236 [2023-12-17:12:01:09:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [14] Learning-rate=0.000180 [2023-12-17:12:01:09:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:01:09:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:01:10:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00014" [2023-12-17:12:01:12:INFO:sockeye.training:__call__] E=14 B=56050 s/sec=842.52 tok/sec=39004.33 u/sec=9.13 ppl=7.859161 [2023-12-17:12:01:14:INFO:sockeye.training:__call__] E=14 B=56100 s/sec=2054.10 tok/sec=96813.89 u/sec=23.22 ppl=7.786871 [2023-12-17:12:01:16:INFO:sockeye.training:__call__] E=14 B=56150 s/sec=2008.83 tok/sec=98455.07 u/sec=23.34 ppl=7.822484 [2023-12-17:12:01:18:INFO:sockeye.training:__call__] E=14 B=56200 s/sec=1864.77 tok/sec=101959.99 u/sec=22.67 ppl=7.816300 [2023-12-17:12:01:20:INFO:sockeye.training:__call__] E=14 B=56250 s/sec=2314.80 tok/sec=95698.65 u/sec=23.80 ppl=7.864802 [2023-12-17:12:01:23:INFO:sockeye.training:__call__] E=14 B=56300 s/sec=2036.59 tok/sec=100820.64 u/sec=23.02 ppl=7.860117 [2023-12-17:12:01:25:INFO:sockeye.training:__call__] E=14 B=56350 s/sec=1979.93 tok/sec=99617.57 u/sec=23.39 ppl=7.854298 [2023-12-17:12:01:27:INFO:sockeye.training:__call__] E=14 B=56400 s/sec=2110.53 tok/sec=99284.06 u/sec=23.22 ppl=7.865973 [2023-12-17:12:01:29:INFO:sockeye.training:__call__] E=14 B=56450 s/sec=2151.85 tok/sec=99200.12 u/sec=23.15 ppl=7.883930 [2023-12-17:12:01:31:INFO:sockeye.training:__call__] E=14 B=56500 s/sec=2158.45 tok/sec=98229.99 u/sec=23.46 ppl=7.892618 [2023-12-17:12:01:33:INFO:sockeye.training:__call__] E=14 B=56550 s/sec=2139.53 tok/sec=99226.74 u/sec=23.18 ppl=7.891532 [2023-12-17:12:01:35:INFO:sockeye.training:__call__] E=14 B=56600 s/sec=2250.39 tok/sec=94914.50 u/sec=23.92 ppl=7.917576 [2023-12-17:12:01:38:INFO:sockeye.training:__call__] E=14 B=56650 s/sec=2102.95 tok/sec=100345.18 u/sec=23.30 ppl=7.928934 [2023-12-17:12:01:40:INFO:sockeye.training:__call__] E=14 B=56700 s/sec=1872.58 tok/sec=100805.24 u/sec=22.81 ppl=7.938649 [2023-12-17:12:01:42:INFO:sockeye.training:__call__] E=14 B=56750 s/sec=2160.80 tok/sec=98443.07 u/sec=23.28 ppl=7.951790 [2023-12-17:12:01:44:INFO:sockeye.training:__call__] E=14 B=56800 s/sec=2032.27 tok/sec=98804.18 u/sec=23.35 ppl=7.957392 [2023-12-17:12:01:46:INFO:sockeye.training:__call__] E=14 B=56850 s/sec=2012.54 tok/sec=103226.62 u/sec=22.50 ppl=7.967709 [2023-12-17:12:01:49:INFO:sockeye.training:__call__] E=14 B=56900 s/sec=1921.06 tok/sec=108906.37 u/sec=21.59 ppl=7.976547 [2023-12-17:12:01:51:INFO:sockeye.training:__call__] E=14 B=56950 s/sec=2153.03 tok/sec=96811.18 u/sec=23.53 ppl=7.989423 [2023-12-17:12:01:53:INFO:sockeye.training:__call__] E=14 B=57000 s/sec=2174.22 tok/sec=99557.37 u/sec=23.03 ppl=8.009395 [2023-12-17:12:01:55:INFO:sockeye.training:__call__] E=14 B=57050 s/sec=2064.98 tok/sec=99416.38 u/sec=23.21 ppl=8.022475 [2023-12-17:12:01:57:INFO:sockeye.training:__call__] E=14 B=57100 s/sec=2043.62 tok/sec=98688.47 u/sec=23.22 ppl=8.034002 [2023-12-17:12:01:59:INFO:sockeye.training:__call__] E=14 B=57150 s/sec=2065.73 tok/sec=98127.97 u/sec=23.60 ppl=8.049851 [2023-12-17:12:02:02:INFO:sockeye.training:__call__] E=14 B=57200 s/sec=1857.81 tok/sec=101977.02 u/sec=22.99 ppl=8.054651 [2023-12-17:12:02:04:INFO:sockeye.training:__call__] E=14 B=57250 s/sec=2260.79 tok/sec=95681.38 u/sec=23.71 ppl=8.076533 [2023-12-17:12:02:06:INFO:sockeye.training:__call__] E=14 B=57300 s/sec=2031.27 tok/sec=100090.19 u/sec=23.25 ppl=8.091497 [2023-12-17:12:02:08:INFO:sockeye.training:__call__] E=14 B=57350 s/sec=1923.44 tok/sec=101733.65 u/sec=22.90 ppl=8.101794 [2023-12-17:12:02:10:INFO:sockeye.training:__call__] E=14 B=57400 s/sec=2145.48 tok/sec=95273.12 u/sec=23.90 ppl=8.117289 [2023-12-17:12:02:12:INFO:sockeye.training:__call__] E=14 B=57450 s/sec=1949.84 tok/sec=102024.75 u/sec=22.48 ppl=8.127179 [2023-12-17:12:02:15:INFO:sockeye.training:__call__] E=14 B=57500 s/sec=1848.09 tok/sec=101355.67 u/sec=22.87 ppl=8.137909 [2023-12-17:12:02:17:INFO:sockeye.training:__call__] E=14 B=57550 s/sec=2123.92 tok/sec=97528.52 u/sec=23.33 ppl=8.152834 [2023-12-17:12:02:19:INFO:sockeye.training:__call__] E=14 B=57600 s/sec=2229.60 tok/sec=97177.64 u/sec=23.30 ppl=8.163797 [2023-12-17:12:02:21:INFO:sockeye.training:__call__] E=14 B=57650 s/sec=2003.54 tok/sec=101195.55 u/sec=23.23 ppl=8.172816 [2023-12-17:12:02:23:INFO:sockeye.training:__call__] E=14 B=57700 s/sec=2143.76 tok/sec=98456.60 u/sec=23.30 ppl=8.183285 [2023-12-17:12:02:25:INFO:sockeye.training:__call__] E=14 B=57750 s/sec=2022.45 tok/sec=99316.68 u/sec=23.32 ppl=8.192515 [2023-12-17:12:02:27:INFO:sockeye.training:__call__] E=14 B=57800 s/sec=1881.87 tok/sec=101236.57 u/sec=22.66 ppl=8.205665 [2023-12-17:12:02:30:INFO:sockeye.training:__call__] E=14 B=57850 s/sec=2182.22 tok/sec=96731.38 u/sec=23.72 ppl=8.217166 [2023-12-17:12:02:32:INFO:sockeye.training:__call__] E=14 B=57900 s/sec=2164.68 tok/sec=97019.73 u/sec=23.49 ppl=8.228428 [2023-12-17:12:02:34:INFO:sockeye.training:__call__] E=14 B=57950 s/sec=2162.92 tok/sec=98959.79 u/sec=22.87 ppl=8.239674 [2023-12-17:12:02:36:INFO:sockeye.training:__call__] E=14 B=58000 s/sec=2056.13 tok/sec=98664.56 u/sec=23.20 ppl=8.250584 [2023-12-17:12:02:38:INFO:sockeye.training:__call__] E=14 B=58050 s/sec=1892.03 tok/sec=102337.09 u/sec=22.78 ppl=8.260418 [2023-12-17:12:02:40:INFO:sockeye.training:__call__] E=14 B=58100 s/sec=2195.10 tok/sec=97663.15 u/sec=23.41 ppl=8.267797 [2023-12-17:12:02:43:INFO:sockeye.training:__call__] E=14 B=58150 s/sec=1818.45 tok/sec=101513.90 u/sec=22.69 ppl=8.273512 [2023-12-17:12:02:45:INFO:sockeye.training:__call__] E=14 B=58200 s/sec=1985.97 tok/sec=99738.41 u/sec=22.53 ppl=8.285084 [2023-12-17:12:02:47:INFO:sockeye.training:__call__] E=14 B=58250 s/sec=1878.56 tok/sec=101565.61 u/sec=22.71 ppl=8.295325 [2023-12-17:12:02:49:INFO:sockeye.training:__call__] E=14 B=58300 s/sec=2110.14 tok/sec=98220.21 u/sec=22.82 ppl=8.305088 [2023-12-17:12:02:51:INFO:sockeye.training:__call__] E=14 B=58350 s/sec=2081.85 tok/sec=98061.83 u/sec=23.15 ppl=8.315913 [2023-12-17:12:02:54:INFO:sockeye.training:__call__] E=14 B=58400 s/sec=2089.45 tok/sec=97851.00 u/sec=22.99 ppl=8.324393 [2023-12-17:12:02:56:INFO:sockeye.training:__call__] E=14 B=58450 s/sec=1889.41 tok/sec=102435.35 u/sec=22.71 ppl=8.329757 [2023-12-17:12:02:58:INFO:sockeye.training:__call__] E=14 B=58500 s/sec=2318.17 tok/sec=92726.86 u/sec=23.99 ppl=8.339025 [2023-12-17:12:03:00:INFO:sockeye.training:__call__] E=14 B=58550 s/sec=2129.03 tok/sec=98834.91 u/sec=23.43 ppl=8.349818 [2023-12-17:12:03:02:INFO:sockeye.training:__call__] E=14 B=58600 s/sec=2030.58 tok/sec=98955.23 u/sec=23.24 ppl=8.358386 [2023-12-17:12:03:04:INFO:sockeye.training:__call__] E=14 B=58650 s/sec=2195.92 tok/sec=99191.58 u/sec=23.22 ppl=8.367518 [2023-12-17:12:03:06:INFO:sockeye.training:__call__] E=14 B=58700 s/sec=1855.65 tok/sec=100402.99 u/sec=22.88 ppl=8.373584 [2023-12-17:12:03:09:INFO:sockeye.training:__call__] E=14 B=58750 s/sec=2086.95 tok/sec=97741.39 u/sec=23.46 ppl=8.381176 [2023-12-17:12:03:11:INFO:sockeye.training:__call__] E=14 B=58800 s/sec=2232.72 tok/sec=96647.77 u/sec=23.69 ppl=8.392150 [2023-12-17:12:03:13:INFO:sockeye.training:__call__] E=14 B=58850 s/sec=1934.00 tok/sec=101118.72 u/sec=22.64 ppl=8.398900 [2023-12-17:12:03:15:INFO:sockeye.training:__call__] E=14 B=58900 s/sec=2094.78 tok/sec=98525.37 u/sec=23.25 ppl=8.409974 [2023-12-17:12:03:17:INFO:sockeye.training:__call__] E=14 B=58950 s/sec=2232.90 tok/sec=96128.42 u/sec=23.65 ppl=8.420638 [2023-12-17:12:03:19:INFO:sockeye.training:__call__] E=14 B=59000 s/sec=1955.29 tok/sec=98454.35 u/sec=23.19 ppl=8.428989 [2023-12-17:12:03:21:INFO:sockeye.training:__call__] E=14 B=59050 s/sec=2177.92 tok/sec=98811.70 u/sec=23.19 ppl=8.439351 [2023-12-17:12:03:24:INFO:sockeye.training:__call__] E=14 B=59100 s/sec=1848.64 tok/sec=103872.56 u/sec=22.70 ppl=8.445466 [2023-12-17:12:03:26:INFO:sockeye.training:__call__] E=14 B=59150 s/sec=1871.83 tok/sec=100781.88 u/sec=23.21 ppl=8.452075 [2023-12-17:12:03:28:INFO:sockeye.training:__call__] E=14 B=59200 s/sec=2058.88 tok/sec=98679.41 u/sec=22.98 ppl=8.461119 [2023-12-17:12:03:30:INFO:sockeye.training:__call__] E=14 B=59250 s/sec=2088.48 tok/sec=98270.85 u/sec=23.39 ppl=8.471642 [2023-12-17:12:03:32:INFO:sockeye.training:__call__] E=14 B=59300 s/sec=2252.88 tok/sec=97007.99 u/sec=23.31 ppl=8.481534 [2023-12-17:12:03:34:INFO:sockeye.training:__call__] E=14 B=59350 s/sec=2252.54 tok/sec=95346.28 u/sec=23.82 ppl=8.490975 [2023-12-17:12:03:37:INFO:sockeye.training:__call__] E=14 B=59400 s/sec=1816.89 tok/sec=105838.47 u/sec=22.05 ppl=8.494151 [2023-12-17:12:03:39:INFO:sockeye.training:__call__] E=14 B=59450 s/sec=1978.21 tok/sec=99648.45 u/sec=23.07 ppl=8.501294 [2023-12-17:12:03:41:INFO:sockeye.training:__call__] E=14 B=59500 s/sec=2160.91 tok/sec=97555.92 u/sec=23.17 ppl=8.511319 [2023-12-17:12:03:43:INFO:sockeye.training:__call__] E=14 B=59550 s/sec=1778.45 tok/sec=102515.59 u/sec=23.06 ppl=8.517742 [2023-12-17:12:03:45:INFO:sockeye.training:__call__] E=14 B=59600 s/sec=2114.11 tok/sec=96212.80 u/sec=23.64 ppl=8.523866 [2023-12-17:12:03:47:INFO:sockeye.training:__call__] E=14 B=59650 s/sec=2210.49 tok/sec=97367.50 u/sec=23.62 ppl=8.533241 [2023-12-17:12:03:49:INFO:sockeye.training:__call__] E=14 B=59700 s/sec=2089.60 tok/sec=97143.94 u/sec=23.49 ppl=8.538282 [2023-12-17:12:03:52:INFO:sockeye.training:__call__] E=14 B=59750 s/sec=2069.52 tok/sec=100694.47 u/sec=23.06 ppl=8.546870 [2023-12-17:12:03:54:INFO:sockeye.training:__call__] E=14 B=59800 s/sec=2161.06 tok/sec=98367.37 u/sec=23.41 ppl=8.553414 [2023-12-17:12:03:56:INFO:sockeye.training:__call__] E=14 B=59850 s/sec=2124.19 tok/sec=100056.23 u/sec=23.25 ppl=8.559715 [2023-12-17:12:03:58:INFO:sockeye.training:__call__] E=14 B=59900 s/sec=2098.85 tok/sec=97986.49 u/sec=23.42 ppl=8.566890 [2023-12-17:12:04:00:INFO:sockeye.training:__call__] E=15 B=59950 s/sec=2047.44 tok/sec=92059.16 u/sec=22.49 ppl=8.562251 [2023-12-17:12:04:02:INFO:sockeye.training:__call__] E=15 B=60000 s/sec=1954.92 tok/sec=101847.52 u/sec=23.01 ppl=8.545496 [2023-12-17:12:04:02:INFO:sockeye.training:_create_checkpoint] Checkpoint [15] Updates=60000 Epoch=15 Samples=5346912 Time-cost=172.743 Updates/sec=23.156 [2023-12-17:12:04:02:INFO:sockeye.training:_create_checkpoint] Checkpoint [15] Train-ppl=8.545496 [2023-12-17:12:04:05:INFO:sockeye.training:_evaluate] Checkpoint [15] Validation-ppl=28.687923 Validation-bleu=0.175246 Validation-chrf=31.592098 Validation-rouge1=0.170929 Validation-rouge2=0.082893 Validation-rougel=0.164011 Validation-length-ratio-mse=1.281912 Validation-ter=93.809327 Validation-avg-sec-per-sent=0.003454 Validation-decode-walltime=1.726968 [2023-12-17:12:04:05:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 10 checkpoints, best so far: 31.983236 [2023-12-17:12:04:05:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [15] Learning-rate=0.000180 [2023-12-17:12:04:06:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:04:06:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:04:06:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00015" [2023-12-17:12:04:08:INFO:sockeye.training:__call__] E=15 B=60050 s/sec=739.91 tok/sec=39952.21 u/sec=9.05 ppl=7.241315 [2023-12-17:12:04:10:INFO:sockeye.training:__call__] E=15 B=60100 s/sec=2211.20 tok/sec=98594.69 u/sec=23.38 ppl=7.337469 [2023-12-17:12:04:12:INFO:sockeye.training:__call__] E=15 B=60150 s/sec=2095.55 tok/sec=97949.20 u/sec=23.68 ppl=7.351622 [2023-12-17:12:04:14:INFO:sockeye.training:__call__] E=15 B=60200 s/sec=1861.98 tok/sec=101904.99 u/sec=22.82 ppl=7.335726 [2023-12-17:12:04:17:INFO:sockeye.training:__call__] E=15 B=60250 s/sec=2316.89 tok/sec=94568.67 u/sec=23.86 ppl=7.383881 [2023-12-17:12:04:19:INFO:sockeye.training:__call__] E=15 B=60300 s/sec=2017.57 tok/sec=97537.13 u/sec=23.57 ppl=7.399723 [2023-12-17:12:04:21:INFO:sockeye.training:__call__] E=15 B=60350 s/sec=1993.67 tok/sec=100099.05 u/sec=22.78 ppl=7.434946 [2023-12-17:12:04:23:INFO:sockeye.training:__call__] E=15 B=60400 s/sec=1943.45 tok/sec=99982.21 u/sec=22.75 ppl=7.450035 [2023-12-17:12:04:25:INFO:sockeye.training:__call__] E=15 B=60450 s/sec=1922.83 tok/sec=100984.09 u/sec=23.25 ppl=7.462857 [2023-12-17:12:04:27:INFO:sockeye.training:__call__] E=15 B=60500 s/sec=2025.89 tok/sec=102769.77 u/sec=22.49 ppl=7.475374 [2023-12-17:12:04:30:INFO:sockeye.training:__call__] E=15 B=60550 s/sec=1910.39 tok/sec=98474.66 u/sec=23.32 ppl=7.482907 [2023-12-17:12:04:32:INFO:sockeye.training:__call__] E=15 B=60600 s/sec=2267.29 tok/sec=96538.66 u/sec=23.31 ppl=7.513902 [2023-12-17:12:04:34:INFO:sockeye.training:__call__] E=15 B=60650 s/sec=2013.94 tok/sec=102225.07 u/sec=22.84 ppl=7.525573 [2023-12-17:12:04:36:INFO:sockeye.training:__call__] E=15 B=60700 s/sec=2213.90 tok/sec=95905.43 u/sec=23.53 ppl=7.545948 [2023-12-17:12:04:38:INFO:sockeye.training:__call__] E=15 B=60750 s/sec=2033.39 tok/sec=100867.70 u/sec=22.98 ppl=7.552270 [2023-12-17:12:04:40:INFO:sockeye.training:__call__] E=15 B=60800 s/sec=2060.76 tok/sec=100490.70 u/sec=23.21 ppl=7.569525 [2023-12-17:12:04:43:INFO:sockeye.training:__call__] E=15 B=60850 s/sec=1798.87 tok/sec=105036.56 u/sec=22.40 ppl=7.571250 [2023-12-17:12:04:45:INFO:sockeye.training:__call__] E=15 B=60900 s/sec=2187.80 tok/sec=95991.65 u/sec=23.58 ppl=7.589757 [2023-12-17:12:04:47:INFO:sockeye.training:__call__] E=15 B=60950 s/sec=2003.88 tok/sec=100075.38 u/sec=23.19 ppl=7.601091 [2023-12-17:12:04:49:INFO:sockeye.training:__call__] E=15 B=61000 s/sec=2055.28 tok/sec=98473.28 u/sec=23.48 ppl=7.607634 [2023-12-17:12:04:51:INFO:sockeye.training:__call__] E=15 B=61050 s/sec=1932.61 tok/sec=101182.77 u/sec=22.83 ppl=7.622161 [2023-12-17:12:04:53:INFO:sockeye.training:__call__] E=15 B=61100 s/sec=2373.24 tok/sec=96318.25 u/sec=23.58 ppl=7.644777 [2023-12-17:12:04:55:INFO:sockeye.training:__call__] E=15 B=61150 s/sec=2151.73 tok/sec=97456.49 u/sec=23.11 ppl=7.657986 [2023-12-17:12:04:58:INFO:sockeye.training:__call__] E=15 B=61200 s/sec=2176.46 tok/sec=97535.27 u/sec=23.25 ppl=7.677365 [2023-12-17:12:05:00:INFO:sockeye.training:__call__] E=15 B=61250 s/sec=2110.29 tok/sec=98030.08 u/sec=23.18 ppl=7.697934 [2023-12-17:12:05:02:INFO:sockeye.training:__call__] E=15 B=61300 s/sec=2016.91 tok/sec=100103.99 u/sec=23.17 ppl=7.707455 [2023-12-17:12:05:04:INFO:sockeye.training:__call__] E=15 B=61350 s/sec=2201.18 tok/sec=97969.24 u/sec=23.28 ppl=7.722849 [2023-12-17:12:05:06:INFO:sockeye.training:__call__] E=15 B=61400 s/sec=1941.49 tok/sec=100043.17 u/sec=22.68 ppl=7.735101 [2023-12-17:12:05:08:INFO:sockeye.training:__call__] E=15 B=61450 s/sec=2208.09 tok/sec=96613.39 u/sec=23.55 ppl=7.750912 [2023-12-17:12:05:11:INFO:sockeye.training:__call__] E=15 B=61500 s/sec=2062.99 tok/sec=99780.08 u/sec=22.74 ppl=7.760770 [2023-12-17:12:05:13:INFO:sockeye.training:__call__] E=15 B=61550 s/sec=1948.03 tok/sec=98403.34 u/sec=23.19 ppl=7.771724 [2023-12-17:12:05:15:INFO:sockeye.training:__call__] E=15 B=61600 s/sec=2175.09 tok/sec=100172.37 u/sec=23.12 ppl=7.784100 [2023-12-17:12:05:17:INFO:sockeye.training:__call__] E=15 B=61650 s/sec=2106.67 tok/sec=97568.95 u/sec=23.51 ppl=7.799408 [2023-12-17:12:05:19:INFO:sockeye.training:__call__] E=15 B=61700 s/sec=1954.36 tok/sec=97970.75 u/sec=23.22 ppl=7.806339 [2023-12-17:12:05:21:INFO:sockeye.training:__call__] E=15 B=61750 s/sec=2190.77 tok/sec=97506.16 u/sec=23.17 ppl=7.821372 [2023-12-17:12:05:23:INFO:sockeye.training:__call__] E=15 B=61800 s/sec=2256.71 tok/sec=94690.34 u/sec=23.83 ppl=7.836796 [2023-12-17:12:05:26:INFO:sockeye.training:__call__] E=15 B=61850 s/sec=2121.82 tok/sec=99067.59 u/sec=23.10 ppl=7.843817 [2023-12-17:12:05:28:INFO:sockeye.training:__call__] E=15 B=61900 s/sec=2187.41 tok/sec=96744.60 u/sec=23.61 ppl=7.855686 [2023-12-17:12:05:30:INFO:sockeye.training:__call__] E=15 B=61950 s/sec=2031.72 tok/sec=101085.25 u/sec=23.00 ppl=7.863180 [2023-12-17:12:05:32:INFO:sockeye.training:__call__] E=15 B=62000 s/sec=2080.50 tok/sec=99036.17 u/sec=23.10 ppl=7.871572 [2023-12-17:12:05:34:INFO:sockeye.training:__call__] E=15 B=62050 s/sec=1947.81 tok/sec=101963.05 u/sec=22.50 ppl=7.880499 [2023-12-17:12:05:36:INFO:sockeye.training:__call__] E=15 B=62100 s/sec=2196.96 tok/sec=98625.54 u/sec=23.19 ppl=7.890215 [2023-12-17:12:05:39:INFO:sockeye.training:__call__] E=15 B=62150 s/sec=2120.47 tok/sec=99804.25 u/sec=23.42 ppl=7.898319 [2023-12-17:12:05:41:INFO:sockeye.training:__call__] E=15 B=62200 s/sec=2140.23 tok/sec=99692.51 u/sec=22.83 ppl=7.909344 [2023-12-17:12:05:43:INFO:sockeye.training:__call__] E=15 B=62250 s/sec=1988.10 tok/sec=98627.54 u/sec=23.14 ppl=7.919041 [2023-12-17:12:05:45:INFO:sockeye.training:__call__] E=15 B=62300 s/sec=1930.16 tok/sec=98235.30 u/sec=23.47 ppl=7.924142 [2023-12-17:12:05:47:INFO:sockeye.training:__call__] E=15 B=62350 s/sec=1910.27 tok/sec=99932.90 u/sec=22.83 ppl=7.929715 [2023-12-17:12:05:49:INFO:sockeye.training:__call__] E=15 B=62400 s/sec=1962.95 tok/sec=98498.78 u/sec=23.37 ppl=7.935829 [2023-12-17:12:05:52:INFO:sockeye.training:__call__] E=15 B=62450 s/sec=2063.08 tok/sec=94671.74 u/sec=23.97 ppl=7.946471 [2023-12-17:12:05:54:INFO:sockeye.training:__call__] E=15 B=62500 s/sec=1972.31 tok/sec=98882.57 u/sec=23.17 ppl=7.952629 [2023-12-17:12:05:56:INFO:sockeye.training:__call__] E=15 B=62550 s/sec=1936.72 tok/sec=98963.26 u/sec=23.32 ppl=7.957210 [2023-12-17:12:05:58:INFO:sockeye.training:__call__] E=15 B=62600 s/sec=1917.50 tok/sec=102680.33 u/sec=23.09 ppl=7.960891 [2023-12-17:12:06:00:INFO:sockeye.training:__call__] E=15 B=62650 s/sec=2066.88 tok/sec=100127.18 u/sec=23.11 ppl=7.970909 [2023-12-17:12:06:02:INFO:sockeye.training:__call__] E=15 B=62700 s/sec=2255.04 tok/sec=97087.15 u/sec=23.49 ppl=7.979869 [2023-12-17:12:06:04:INFO:sockeye.training:__call__] E=15 B=62750 s/sec=1903.69 tok/sec=98722.05 u/sec=23.42 ppl=7.987676 [2023-12-17:12:06:07:INFO:sockeye.training:__call__] E=15 B=62800 s/sec=2213.37 tok/sec=95809.80 u/sec=23.49 ppl=7.998996 [2023-12-17:12:06:09:INFO:sockeye.training:__call__] E=15 B=62850 s/sec=2159.81 tok/sec=97961.27 u/sec=23.48 ppl=8.008504 [2023-12-17:12:06:11:INFO:sockeye.training:__call__] E=15 B=62900 s/sec=1858.62 tok/sec=102012.79 u/sec=22.78 ppl=8.016146 [2023-12-17:12:06:13:INFO:sockeye.training:__call__] E=15 B=62950 s/sec=2008.76 tok/sec=100066.18 u/sec=22.79 ppl=8.023006 [2023-12-17:12:06:15:INFO:sockeye.training:__call__] E=15 B=63000 s/sec=2238.60 tok/sec=97352.66 u/sec=23.55 ppl=8.030843 [2023-12-17:12:06:17:INFO:sockeye.training:__call__] E=15 B=63050 s/sec=1827.96 tok/sec=102629.41 u/sec=22.90 ppl=8.034061 [2023-12-17:12:06:19:INFO:sockeye.training:__call__] E=15 B=63100 s/sec=2378.27 tok/sec=94949.76 u/sec=23.59 ppl=8.044457 [2023-12-17:12:06:22:INFO:sockeye.training:__call__] E=15 B=63150 s/sec=1874.01 tok/sec=103913.62 u/sec=21.77 ppl=8.051647 [2023-12-17:12:06:24:INFO:sockeye.training:__call__] E=15 B=63200 s/sec=1887.70 tok/sec=107592.14 u/sec=21.30 ppl=8.057205 [2023-12-17:12:06:26:INFO:sockeye.training:__call__] E=15 B=63250 s/sec=2110.75 tok/sec=99420.21 u/sec=23.14 ppl=8.066729 [2023-12-17:12:06:28:INFO:sockeye.training:__call__] E=15 B=63300 s/sec=2084.32 tok/sec=99782.92 u/sec=22.98 ppl=8.075161 [2023-12-17:12:06:31:INFO:sockeye.training:__call__] E=15 B=63350 s/sec=2123.87 tok/sec=98500.00 u/sec=23.21 ppl=8.083577 [2023-12-17:12:06:33:INFO:sockeye.training:__call__] E=15 B=63400 s/sec=1995.24 tok/sec=99858.80 u/sec=23.31 ppl=8.091953 [2023-12-17:12:06:35:INFO:sockeye.training:__call__] E=15 B=63450 s/sec=2181.58 tok/sec=98345.33 u/sec=23.15 ppl=8.101842 [2023-12-17:12:06:37:INFO:sockeye.training:__call__] E=15 B=63500 s/sec=2209.42 tok/sec=97244.43 u/sec=23.40 ppl=8.111488 [2023-12-17:12:06:39:INFO:sockeye.training:__call__] E=15 B=63550 s/sec=2074.31 tok/sec=98201.32 u/sec=23.19 ppl=8.119740 [2023-12-17:12:06:41:INFO:sockeye.training:__call__] E=15 B=63600 s/sec=2310.29 tok/sec=94036.13 u/sec=23.95 ppl=8.129783 [2023-12-17:12:06:43:INFO:sockeye.training:__call__] E=15 B=63650 s/sec=2005.50 tok/sec=99684.99 u/sec=23.04 ppl=8.136766 [2023-12-17:12:06:46:INFO:sockeye.training:__call__] E=15 B=63700 s/sec=2162.38 tok/sec=100780.48 u/sec=22.64 ppl=8.144803 [2023-12-17:12:06:48:INFO:sockeye.training:__call__] E=15 B=63750 s/sec=1976.03 tok/sec=100187.33 u/sec=22.91 ppl=8.150624 [2023-12-17:12:06:50:INFO:sockeye.training:__call__] E=15 B=63800 s/sec=2152.50 tok/sec=94572.30 u/sec=23.90 ppl=8.158553 [2023-12-17:12:06:52:INFO:sockeye.training:__call__] E=15 B=63850 s/sec=1992.86 tok/sec=101876.95 u/sec=22.81 ppl=8.164673 [2023-12-17:12:06:54:INFO:sockeye.training:__call__] E=15 B=63900 s/sec=2166.79 tok/sec=94927.60 u/sec=23.80 ppl=8.170475 [2023-12-17:12:06:57:INFO:sockeye.training:__call__] E=16 B=63950 s/sec=1829.39 tok/sec=97426.46 u/sec=22.03 ppl=8.162186 [2023-12-17:12:06:59:INFO:sockeye.training:__call__] E=16 B=64000 s/sec=2014.72 tok/sec=99556.10 u/sec=23.19 ppl=8.147151 [2023-12-17:12:06:59:INFO:sockeye.training:_create_checkpoint] Checkpoint [16] Updates=64000 Epoch=16 Samples=5703512 Time-cost=172.853 Updates/sec=23.141 [2023-12-17:12:06:59:INFO:sockeye.training:_create_checkpoint] Checkpoint [16] Train-ppl=8.147151 [2023-12-17:12:07:01:INFO:sockeye.training:_evaluate] Checkpoint [16] Validation-ppl=29.548111 Validation-bleu=0.171251 Validation-chrf=31.124228 Validation-rouge1=0.153663 Validation-rouge2=0.076671 Validation-rougel=0.147653 Validation-length-ratio-mse=1.300916 Validation-ter=96.987206 Validation-avg-sec-per-sent=0.003525 Validation-decode-walltime=1.762334 [2023-12-17:12:07:01:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 11 checkpoints, best so far: 31.983236 [2023-12-17:12:07:01:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [16] Learning-rate=0.000180 [2023-12-17:12:07:02:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:07:02:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=3, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:07:02:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00016" [2023-12-17:12:07:04:INFO:sockeye.training:__call__] E=16 B=64050 s/sec=767.37 tok/sec=39434.83 u/sec=9.03 ppl=7.019364 [2023-12-17:12:07:06:INFO:sockeye.training:__call__] E=16 B=64100 s/sec=1952.42 tok/sec=101891.90 u/sec=22.89 ppl=7.034194 [2023-12-17:12:07:09:INFO:sockeye.training:__call__] E=16 B=64150 s/sec=2124.02 tok/sec=98417.97 u/sec=23.21 ppl=7.083818 [2023-12-17:12:07:11:INFO:sockeye.training:__call__] E=16 B=64200 s/sec=1941.06 tok/sec=99658.34 u/sec=22.93 ppl=7.075590 [2023-12-17:12:07:13:INFO:sockeye.training:__call__] E=16 B=64250 s/sec=1964.15 tok/sec=98229.73 u/sec=23.38 ppl=7.087249 [2023-12-17:12:07:15:INFO:sockeye.training:__call__] E=16 B=64300 s/sec=1828.93 tok/sec=102655.35 u/sec=22.95 ppl=7.092815 [2023-12-17:12:07:17:INFO:sockeye.training:__call__] E=16 B=64350 s/sec=1970.41 tok/sec=101463.51 u/sec=22.60 ppl=7.107643 [2023-12-17:12:07:19:INFO:sockeye.training:__call__] E=16 B=64400 s/sec=2098.94 tok/sec=97885.83 u/sec=23.30 ppl=7.118538 [2023-12-17:12:07:22:INFO:sockeye.training:__call__] E=16 B=64450 s/sec=1893.16 tok/sec=102369.24 u/sec=22.80 ppl=7.133476 [2023-12-17:12:07:24:INFO:sockeye.training:__call__] E=16 B=64500 s/sec=2035.96 tok/sec=100669.13 u/sec=22.76 ppl=7.156145 [2023-12-17:12:07:26:INFO:sockeye.training:__call__] E=16 B=64550 s/sec=2158.97 tok/sec=99628.78 u/sec=22.99 ppl=7.182879 [2023-12-17:12:07:28:INFO:sockeye.training:__call__] E=16 B=64600 s/sec=2126.89 tok/sec=98224.85 u/sec=22.88 ppl=7.199760 [2023-12-17:12:07:30:INFO:sockeye.training:__call__] E=16 B=64650 s/sec=2343.78 tok/sec=96124.92 u/sec=23.48 ppl=7.221750 [2023-12-17:12:07:32:INFO:sockeye.training:__call__] E=16 B=64700 s/sec=1960.82 tok/sec=100846.49 u/sec=22.95 ppl=7.229830 [2023-12-17:12:07:35:INFO:sockeye.training:__call__] E=16 B=64750 s/sec=1980.93 tok/sec=98373.83 u/sec=23.36 ppl=7.236368 [2023-12-17:12:07:37:INFO:sockeye.training:__call__] E=16 B=64800 s/sec=1992.43 tok/sec=99503.37 u/sec=23.06 ppl=7.242851 [2023-12-17:12:07:39:INFO:sockeye.training:__call__] E=16 B=64850 s/sec=1910.93 tok/sec=104190.66 u/sec=22.49 ppl=7.250413 [2023-12-17:12:07:41:INFO:sockeye.training:__call__] E=16 B=64900 s/sec=2227.77 tok/sec=95820.79 u/sec=23.72 ppl=7.273376 [2023-12-17:12:07:43:INFO:sockeye.training:__call__] E=16 B=64950 s/sec=2146.66 tok/sec=99605.04 u/sec=22.93 ppl=7.291188 [2023-12-17:12:07:45:INFO:sockeye.training:__call__] E=16 B=65000 s/sec=2094.55 tok/sec=99290.82 u/sec=23.21 ppl=7.306971 [2023-12-17:12:07:48:INFO:sockeye.training:__call__] E=16 B=65050 s/sec=2031.23 tok/sec=95883.19 u/sec=23.38 ppl=7.319520 [2023-12-17:12:07:50:INFO:sockeye.training:__call__] E=16 B=65100 s/sec=2070.91 tok/sec=97407.35 u/sec=23.28 ppl=7.328948 [2023-12-17:12:07:52:INFO:sockeye.training:__call__] E=16 B=65150 s/sec=1988.79 tok/sec=100952.36 u/sec=23.06 ppl=7.340756 [2023-12-17:12:07:54:INFO:sockeye.training:__call__] E=16 B=65200 s/sec=1888.41 tok/sec=99613.66 u/sec=23.05 ppl=7.358647 [2023-12-17:12:07:56:INFO:sockeye.training:__call__] E=16 B=65250 s/sec=1834.63 tok/sec=98237.04 u/sec=23.45 ppl=7.359930 [2023-12-17:12:07:58:INFO:sockeye.training:__call__] E=16 B=65300 s/sec=2168.56 tok/sec=99470.56 u/sec=23.29 ppl=7.373586 [2023-12-17:12:08:01:INFO:sockeye.training:__call__] E=16 B=65350 s/sec=1831.21 tok/sec=102978.45 u/sec=22.44 ppl=7.379293 [2023-12-17:12:08:03:INFO:sockeye.training:__call__] E=16 B=65400 s/sec=2034.72 tok/sec=98877.59 u/sec=23.08 ppl=7.393728 [2023-12-17:12:08:05:INFO:sockeye.training:__call__] E=16 B=65450 s/sec=2076.39 tok/sec=99637.23 u/sec=23.13 ppl=7.404337 [2023-12-17:12:08:07:INFO:sockeye.training:__call__] E=16 B=65500 s/sec=2433.08 tok/sec=93975.62 u/sec=24.21 ppl=7.422916 [2023-12-17:12:08:09:INFO:sockeye.training:__call__] E=16 B=65550 s/sec=2170.99 tok/sec=97675.53 u/sec=23.52 ppl=7.434070 [2023-12-17:12:08:11:INFO:sockeye.training:__call__] E=16 B=65600 s/sec=2091.14 tok/sec=105839.78 u/sec=21.78 ppl=7.442864 [2023-12-17:12:08:13:INFO:sockeye.training:__call__] E=16 B=65650 s/sec=2244.23 tok/sec=93966.51 u/sec=23.94 ppl=7.456539 [2023-12-17:12:08:16:INFO:sockeye.training:__call__] E=16 B=65700 s/sec=2077.76 tok/sec=100408.08 u/sec=22.94 ppl=7.470398 [2023-12-17:12:08:18:INFO:sockeye.training:__call__] E=16 B=65750 s/sec=2048.66 tok/sec=97260.75 u/sec=23.32 ppl=7.477808 [2023-12-17:12:08:20:INFO:sockeye.training:__call__] E=16 B=65800 s/sec=2180.72 tok/sec=97378.04 u/sec=23.46 ppl=7.488259 [2023-12-17:12:08:22:INFO:sockeye.training:__call__] E=16 B=65850 s/sec=2098.96 tok/sec=98610.41 u/sec=23.22 ppl=7.497932 [2023-12-17:12:08:24:INFO:sockeye.training:__call__] E=16 B=65900 s/sec=2420.22 tok/sec=92264.13 u/sec=24.32 ppl=7.512657 [2023-12-17:12:08:26:INFO:sockeye.training:__call__] E=16 B=65950 s/sec=1988.30 tok/sec=101182.59 u/sec=23.01 ppl=7.521607 [2023-12-17:12:08:28:INFO:sockeye.training:__call__] E=16 B=66000 s/sec=2289.52 tok/sec=95483.85 u/sec=24.01 ppl=7.537213 [2023-12-17:12:08:31:INFO:sockeye.training:__call__] E=16 B=66050 s/sec=2260.78 tok/sec=95693.79 u/sec=23.63 ppl=7.547281 [2023-12-17:12:08:33:INFO:sockeye.training:__call__] E=16 B=66100 s/sec=2331.45 tok/sec=96025.78 u/sec=23.50 ppl=7.560809 [2023-12-17:12:08:35:INFO:sockeye.training:__call__] E=16 B=66150 s/sec=2041.29 tok/sec=102202.53 u/sec=22.74 ppl=7.568015 [2023-12-17:12:08:37:INFO:sockeye.training:__call__] E=16 B=66200 s/sec=2183.75 tok/sec=97920.98 u/sec=23.13 ppl=7.579785 [2023-12-17:12:08:39:INFO:sockeye.training:__call__] E=16 B=66250 s/sec=2350.75 tok/sec=95090.11 u/sec=23.66 ppl=7.590375 [2023-12-17:12:08:41:INFO:sockeye.training:__call__] E=16 B=66300 s/sec=2087.72 tok/sec=98460.30 u/sec=23.18 ppl=7.599432 [2023-12-17:12:08:43:INFO:sockeye.training:__call__] E=16 B=66350 s/sec=1923.19 tok/sec=101037.60 u/sec=23.03 ppl=7.607702 [2023-12-17:12:08:46:INFO:sockeye.training:__call__] E=16 B=66400 s/sec=1933.58 tok/sec=100257.69 u/sec=22.55 ppl=7.617915 [2023-12-17:12:08:48:INFO:sockeye.training:__call__] E=16 B=66450 s/sec=2157.03 tok/sec=96347.52 u/sec=23.41 ppl=7.624782 [2023-12-17:12:08:50:INFO:sockeye.training:__call__] E=16 B=66500 s/sec=2138.47 tok/sec=95681.16 u/sec=23.87 ppl=7.634018 [2023-12-17:12:08:52:INFO:sockeye.training:__call__] E=16 B=66550 s/sec=2162.90 tok/sec=98334.22 u/sec=23.67 ppl=7.644498 [2023-12-17:12:08:54:INFO:sockeye.training:__call__] E=16 B=66600 s/sec=2013.62 tok/sec=99383.50 u/sec=23.44 ppl=7.654546 [2023-12-17:12:08:56:INFO:sockeye.training:__call__] E=16 B=66650 s/sec=1927.04 tok/sec=101252.67 u/sec=23.03 ppl=7.659589 [2023-12-17:12:08:59:INFO:sockeye.training:__call__] E=16 B=66700 s/sec=1945.79 tok/sec=100539.60 u/sec=22.77 ppl=7.664010 [2023-12-17:12:09:01:INFO:sockeye.training:__call__] E=16 B=66750 s/sec=2163.07 tok/sec=97361.12 u/sec=23.72 ppl=7.670545 [2023-12-17:12:09:03:INFO:sockeye.training:__call__] E=16 B=66800 s/sec=1997.24 tok/sec=101617.16 u/sec=22.57 ppl=7.679178 [2023-12-17:12:09:05:INFO:sockeye.training:__call__] E=16 B=66850 s/sec=1904.93 tok/sec=97903.91 u/sec=23.39 ppl=7.685310 [2023-12-17:12:09:07:INFO:sockeye.training:__call__] E=16 B=66900 s/sec=2016.37 tok/sec=99667.17 u/sec=23.21 ppl=7.693967 [2023-12-17:12:09:09:INFO:sockeye.training:__call__] E=16 B=66950 s/sec=2106.58 tok/sec=97096.19 u/sec=23.26 ppl=7.702932 [2023-12-17:12:09:11:INFO:sockeye.training:__call__] E=16 B=67000 s/sec=2176.62 tok/sec=98265.43 u/sec=23.33 ppl=7.710392 [2023-12-17:12:09:14:INFO:sockeye.training:__call__] E=16 B=67050 s/sec=2189.41 tok/sec=94960.89 u/sec=23.84 ppl=7.717828 [2023-12-17:12:09:16:INFO:sockeye.training:__call__] E=16 B=67100 s/sec=2037.05 tok/sec=98109.56 u/sec=23.53 ppl=7.724927 [2023-12-17:12:09:18:INFO:sockeye.training:__call__] E=16 B=67150 s/sec=1941.73 tok/sec=99638.90 u/sec=22.85 ppl=7.731004 [2023-12-17:12:09:20:INFO:sockeye.training:__call__] E=16 B=67200 s/sec=1973.21 tok/sec=101634.82 u/sec=23.01 ppl=7.738429 [2023-12-17:12:09:22:INFO:sockeye.training:__call__] E=16 B=67250 s/sec=2055.54 tok/sec=103273.77 u/sec=22.50 ppl=7.744302 [2023-12-17:12:09:24:INFO:sockeye.training:__call__] E=16 B=67300 s/sec=2227.05 tok/sec=96874.63 u/sec=23.87 ppl=7.753189 [2023-12-17:12:09:27:INFO:sockeye.training:__call__] E=16 B=67350 s/sec=2091.38 tok/sec=100708.59 u/sec=22.89 ppl=7.761463 [2023-12-17:12:09:29:INFO:sockeye.training:__call__] E=16 B=67400 s/sec=2107.64 tok/sec=98361.42 u/sec=23.31 ppl=7.769348 [2023-12-17:12:09:31:INFO:sockeye.training:__call__] E=16 B=67450 s/sec=2022.99 tok/sec=100334.15 u/sec=23.16 ppl=7.774857 [2023-12-17:12:09:33:INFO:sockeye.training:__call__] E=16 B=67500 s/sec=2127.52 tok/sec=99150.54 u/sec=23.21 ppl=7.782552 [2023-12-17:12:09:35:INFO:sockeye.training:__call__] E=16 B=67550 s/sec=1981.86 tok/sec=99520.49 u/sec=23.02 ppl=7.791894 [2023-12-17:12:09:37:INFO:sockeye.training:__call__] E=16 B=67600 s/sec=1953.80 tok/sec=101162.05 u/sec=22.70 ppl=7.798607 [2023-12-17:12:09:40:INFO:sockeye.training:__call__] E=16 B=67650 s/sec=2071.23 tok/sec=100323.28 u/sec=22.79 ppl=7.805892 [2023-12-17:12:09:42:INFO:sockeye.training:__call__] E=16 B=67700 s/sec=2130.86 tok/sec=98162.37 u/sec=23.49 ppl=7.813400 [2023-12-17:12:09:44:INFO:sockeye.training:__call__] E=16 B=67750 s/sec=1861.97 tok/sec=102411.76 u/sec=22.68 ppl=7.818187 [2023-12-17:12:09:46:INFO:sockeye.training:__call__] E=16 B=67800 s/sec=2110.02 tok/sec=97399.63 u/sec=23.51 ppl=7.823147 [2023-12-17:12:09:48:INFO:sockeye.training:__call__] E=16 B=67850 s/sec=2054.89 tok/sec=98930.81 u/sec=23.14 ppl=7.829152 [2023-12-17:12:09:50:INFO:sockeye.training:__call__] E=16 B=67900 s/sec=1957.99 tok/sec=102386.33 u/sec=22.87 ppl=7.834840 [2023-12-17:12:09:53:INFO:sockeye.training:__call__] E=17 B=67950 s/sec=1751.96 tok/sec=99296.59 u/sec=22.08 ppl=7.827147 [2023-12-17:12:09:55:INFO:sockeye.training:__call__] E=17 B=68000 s/sec=2085.90 tok/sec=99441.65 u/sec=23.16 ppl=7.814129 [2023-12-17:12:09:55:INFO:sockeye.training:_create_checkpoint] Checkpoint [17] Updates=68000 Epoch=17 Samples=6059920 Time-cost=172.734 Updates/sec=23.157 [2023-12-17:12:09:55:INFO:sockeye.training:_create_checkpoint] Checkpoint [17] Train-ppl=7.814129 [2023-12-17:12:09:57:INFO:sockeye.training:_evaluate] Checkpoint [17] Validation-ppl=30.078653 Validation-bleu=0.157769 Validation-chrf=30.090398 Validation-rouge1=0.160824 Validation-rouge2=0.079840 Validation-rougel=0.154389 Validation-length-ratio-mse=1.305848 Validation-ter=97.812629 Validation-avg-sec-per-sent=0.003291 Validation-decode-walltime=1.645558 [2023-12-17:12:09:57:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 12 checkpoints, best so far: 31.983236 [2023-12-17:12:09:57:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [17] Learning-rate=0.000180 [2023-12-17:12:09:58:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:09:58:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=4, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:09:58:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00017" [2023-12-17:12:10:00:INFO:sockeye.training:__call__] E=17 B=68050 s/sec=752.20 tok/sec=39812.21 u/sec=9.16 ppl=6.731433 [2023-12-17:12:10:02:INFO:sockeye.training:__call__] E=17 B=68100 s/sec=2284.12 tok/sec=96023.98 u/sec=23.64 ppl=6.839886 [2023-12-17:12:10:04:INFO:sockeye.training:__call__] E=17 B=68150 s/sec=2154.99 tok/sec=99511.80 u/sec=23.42 ppl=6.870842 [2023-12-17:12:10:07:INFO:sockeye.training:__call__] E=17 B=68200 s/sec=2113.20 tok/sec=100811.93 u/sec=23.13 ppl=6.861070 [2023-12-17:12:10:09:INFO:sockeye.training:__call__] E=17 B=68250 s/sec=1926.30 tok/sec=101343.74 u/sec=22.98 ppl=6.857021 [2023-12-17:12:10:11:INFO:sockeye.training:__call__] E=17 B=68300 s/sec=2265.85 tok/sec=95769.59 u/sec=23.88 ppl=6.877688 [2023-12-17:12:10:13:INFO:sockeye.training:__call__] E=17 B=68350 s/sec=1922.51 tok/sec=100851.47 u/sec=23.33 ppl=6.885332 [2023-12-17:12:10:15:INFO:sockeye.training:__call__] E=17 B=68400 s/sec=1946.54 tok/sec=102974.76 u/sec=22.66 ppl=6.897742 [2023-12-17:12:10:17:INFO:sockeye.training:__call__] E=17 B=68450 s/sec=2331.34 tok/sec=96483.88 u/sec=23.81 ppl=6.920167 [2023-12-17:12:10:20:INFO:sockeye.training:__call__] E=17 B=68500 s/sec=2091.63 tok/sec=97817.24 u/sec=23.18 ppl=6.942617 [2023-12-17:12:10:22:INFO:sockeye.training:__call__] E=17 B=68550 s/sec=1761.92 tok/sec=104827.37 u/sec=22.38 ppl=6.940193 [2023-12-17:12:10:24:INFO:sockeye.training:__call__] E=17 B=68600 s/sec=1756.18 tok/sec=102812.33 u/sec=23.11 ppl=6.942563 [2023-12-17:12:10:26:INFO:sockeye.training:__call__] E=17 B=68650 s/sec=2229.39 tok/sec=98077.76 u/sec=23.82 ppl=6.964135 [2023-12-17:12:10:28:INFO:sockeye.training:__call__] E=17 B=68700 s/sec=2197.34 tok/sec=95182.42 u/sec=23.93 ppl=6.982443 [2023-12-17:12:10:30:INFO:sockeye.training:__call__] E=17 B=68750 s/sec=2329.43 tok/sec=95896.11 u/sec=23.87 ppl=7.001259 [2023-12-17:12:10:32:INFO:sockeye.training:__call__] E=17 B=68800 s/sec=1879.83 tok/sec=103138.49 u/sec=22.51 ppl=7.003365 [2023-12-17:12:10:35:INFO:sockeye.training:__call__] E=17 B=68850 s/sec=1964.86 tok/sec=98926.63 u/sec=22.25 ppl=7.014911 [2023-12-17:12:10:37:INFO:sockeye.training:__call__] E=17 B=68900 s/sec=2273.22 tok/sec=97336.71 u/sec=23.84 ppl=7.036502 [2023-12-17:12:10:39:INFO:sockeye.training:__call__] E=17 B=68950 s/sec=2280.97 tok/sec=95060.99 u/sec=24.08 ppl=7.058026 [2023-12-17:12:10:41:INFO:sockeye.training:__call__] E=17 B=69000 s/sec=1974.02 tok/sec=102486.29 u/sec=23.15 ppl=7.067393 [2023-12-17:12:10:43:INFO:sockeye.training:__call__] E=17 B=69050 s/sec=2061.66 tok/sec=99706.49 u/sec=23.34 ppl=7.085312 [2023-12-17:12:10:45:INFO:sockeye.training:__call__] E=17 B=69100 s/sec=1957.86 tok/sec=102013.53 u/sec=22.83 ppl=7.091701 [2023-12-17:12:10:48:INFO:sockeye.training:__call__] E=17 B=69150 s/sec=2060.48 tok/sec=100859.63 u/sec=23.16 ppl=7.101484 [2023-12-17:12:10:50:INFO:sockeye.training:__call__] E=17 B=69200 s/sec=2045.36 tok/sec=101241.46 u/sec=23.24 ppl=7.108592 [2023-12-17:12:10:52:INFO:sockeye.training:__call__] E=17 B=69250 s/sec=1975.27 tok/sec=102728.82 u/sec=22.90 ppl=7.117612 [2023-12-17:12:10:54:INFO:sockeye.training:__call__] E=17 B=69300 s/sec=2027.66 tok/sec=98762.57 u/sec=22.88 ppl=7.127108 [2023-12-17:12:10:56:INFO:sockeye.training:__call__] E=17 B=69350 s/sec=2007.06 tok/sec=98293.33 u/sec=23.49 ppl=7.137926 [2023-12-17:12:10:58:INFO:sockeye.training:__call__] E=17 B=69400 s/sec=2115.68 tok/sec=99065.49 u/sec=23.70 ppl=7.149543 [2023-12-17:12:11:00:INFO:sockeye.training:__call__] E=17 B=69450 s/sec=1971.99 tok/sec=102879.82 u/sec=22.87 ppl=7.161744 [2023-12-17:12:11:03:INFO:sockeye.training:__call__] E=17 B=69500 s/sec=2213.87 tok/sec=102990.02 u/sec=22.65 ppl=7.174173 [2023-12-17:12:11:05:INFO:sockeye.training:__call__] E=17 B=69550 s/sec=2199.36 tok/sec=94643.98 u/sec=23.58 ppl=7.186319 [2023-12-17:12:11:07:INFO:sockeye.training:__call__] E=17 B=69600 s/sec=1938.01 tok/sec=101605.58 u/sec=22.73 ppl=7.194227 [2023-12-17:12:11:09:INFO:sockeye.training:__call__] E=17 B=69650 s/sec=2470.40 tok/sec=94349.34 u/sec=23.90 ppl=7.208120 [2023-12-17:12:11:11:INFO:sockeye.training:__call__] E=17 B=69700 s/sec=2289.22 tok/sec=95395.45 u/sec=23.73 ppl=7.222155 [2023-12-17:12:11:13:INFO:sockeye.training:__call__] E=17 B=69750 s/sec=2032.32 tok/sec=100477.44 u/sec=23.57 ppl=7.232366 [2023-12-17:12:11:15:INFO:sockeye.training:__call__] E=17 B=69800 s/sec=1862.63 tok/sec=101888.63 u/sec=23.33 ppl=7.239018 [2023-12-17:12:11:18:INFO:sockeye.training:__call__] E=17 B=69850 s/sec=2077.24 tok/sec=97840.50 u/sec=23.91 ppl=7.247012 [2023-12-17:12:11:20:INFO:sockeye.training:__call__] E=17 B=69900 s/sec=1994.95 tok/sec=100495.34 u/sec=23.13 ppl=7.255303 [2023-12-17:12:11:22:INFO:sockeye.training:__call__] E=17 B=69950 s/sec=2082.90 tok/sec=99564.80 u/sec=23.12 ppl=7.261468 [2023-12-17:12:11:24:INFO:sockeye.training:__call__] E=17 B=70000 s/sec=1959.30 tok/sec=101002.92 u/sec=22.55 ppl=7.267182 [2023-12-17:12:11:26:INFO:sockeye.training:__call__] E=17 B=70050 s/sec=2261.36 tok/sec=96085.38 u/sec=23.40 ppl=7.280473 [2023-12-17:12:11:28:INFO:sockeye.training:__call__] E=17 B=70100 s/sec=2163.95 tok/sec=98973.93 u/sec=23.04 ppl=7.289965 [2023-12-17:12:11:31:INFO:sockeye.training:__call__] E=17 B=70150 s/sec=2126.19 tok/sec=98721.24 u/sec=23.48 ppl=7.301354 [2023-12-17:12:11:33:INFO:sockeye.training:__call__] E=17 B=70200 s/sec=2332.77 tok/sec=97091.67 u/sec=23.63 ppl=7.312308 [2023-12-17:12:11:35:INFO:sockeye.training:__call__] E=17 B=70250 s/sec=2243.41 tok/sec=94555.42 u/sec=24.13 ppl=7.323669 [2023-12-17:12:11:37:INFO:sockeye.training:__call__] E=17 B=70300 s/sec=1802.46 tok/sec=104600.56 u/sec=22.71 ppl=7.329419 [2023-12-17:12:11:39:INFO:sockeye.training:__call__] E=17 B=70350 s/sec=2109.64 tok/sec=99515.64 u/sec=23.13 ppl=7.339555 [2023-12-17:12:11:41:INFO:sockeye.training:__call__] E=17 B=70400 s/sec=2046.35 tok/sec=100256.41 u/sec=23.34 ppl=7.346139 [2023-12-17:12:11:43:INFO:sockeye.training:__call__] E=17 B=70450 s/sec=1944.31 tok/sec=104018.99 u/sec=22.22 ppl=7.350725 [2023-12-17:12:11:46:INFO:sockeye.training:__call__] E=17 B=70500 s/sec=1914.73 tok/sec=101528.87 u/sec=23.06 ppl=7.356161 [2023-12-17:12:11:48:INFO:sockeye.training:__call__] E=17 B=70550 s/sec=2240.91 tok/sec=95662.52 u/sec=23.54 ppl=7.366147 [2023-12-17:12:11:50:INFO:sockeye.training:__call__] E=17 B=70600 s/sec=1817.74 tok/sec=102552.93 u/sec=22.81 ppl=7.372710 [2023-12-17:12:11:52:INFO:sockeye.training:__call__] E=17 B=70650 s/sec=2019.47 tok/sec=98576.33 u/sec=22.91 ppl=7.379375 [2023-12-17:12:11:54:INFO:sockeye.training:__call__] E=17 B=70700 s/sec=2237.87 tok/sec=97676.69 u/sec=23.27 ppl=7.390947 [2023-12-17:12:11:56:INFO:sockeye.training:__call__] E=17 B=70750 s/sec=2097.17 tok/sec=98013.69 u/sec=23.53 ppl=7.397752 [2023-12-17:12:11:59:INFO:sockeye.training:__call__] E=17 B=70800 s/sec=2147.62 tok/sec=98575.42 u/sec=23.18 ppl=7.407284 [2023-12-17:12:12:01:INFO:sockeye.training:__call__] E=17 B=70850 s/sec=1983.97 tok/sec=101912.06 u/sec=22.79 ppl=7.414583 [2023-12-17:12:12:03:INFO:sockeye.training:__call__] E=17 B=70900 s/sec=1785.00 tok/sec=103941.16 u/sec=22.54 ppl=7.419855 [2023-12-17:12:12:05:INFO:sockeye.training:__call__] E=17 B=70950 s/sec=2157.92 tok/sec=97462.29 u/sec=23.66 ppl=7.428399 [2023-12-17:12:12:07:INFO:sockeye.training:__call__] E=17 B=71000 s/sec=2011.67 tok/sec=101051.96 u/sec=23.24 ppl=7.435676 [2023-12-17:12:12:09:INFO:sockeye.training:__call__] E=17 B=71050 s/sec=2210.32 tok/sec=97663.50 u/sec=23.70 ppl=7.443737 [2023-12-17:12:12:11:INFO:sockeye.training:__call__] E=17 B=71100 s/sec=2370.30 tok/sec=94318.63 u/sec=24.09 ppl=7.450865 [2023-12-17:12:12:14:INFO:sockeye.training:__call__] E=17 B=71150 s/sec=1868.83 tok/sec=102363.38 u/sec=22.95 ppl=7.456211 [2023-12-17:12:12:16:INFO:sockeye.training:__call__] E=17 B=71200 s/sec=2122.81 tok/sec=99828.36 u/sec=23.40 ppl=7.464293 [2023-12-17:12:12:18:INFO:sockeye.training:__call__] E=17 B=71250 s/sec=2259.43 tok/sec=99116.76 u/sec=23.30 ppl=7.473204 [2023-12-17:12:12:20:INFO:sockeye.training:__call__] E=17 B=71300 s/sec=2025.19 tok/sec=98462.05 u/sec=23.31 ppl=7.479381 [2023-12-17:12:12:22:INFO:sockeye.training:__call__] E=17 B=71350 s/sec=2049.45 tok/sec=99966.21 u/sec=23.04 ppl=7.485516 [2023-12-17:12:12:24:INFO:sockeye.training:__call__] E=17 B=71400 s/sec=2214.91 tok/sec=96927.64 u/sec=23.58 ppl=7.493100 [2023-12-17:12:12:26:INFO:sockeye.training:__call__] E=17 B=71450 s/sec=2203.86 tok/sec=98899.22 u/sec=23.19 ppl=7.500657 [2023-12-17:12:12:29:INFO:sockeye.training:__call__] E=17 B=71500 s/sec=1827.60 tok/sec=103711.54 u/sec=22.71 ppl=7.505533 [2023-12-17:12:12:31:INFO:sockeye.training:__call__] E=17 B=71550 s/sec=1733.37 tok/sec=103616.13 u/sec=22.34 ppl=7.509906 [2023-12-17:12:12:33:INFO:sockeye.training:__call__] E=17 B=71600 s/sec=2033.00 tok/sec=99728.92 u/sec=23.27 ppl=7.516208 [2023-12-17:12:12:35:INFO:sockeye.training:__call__] E=17 B=71650 s/sec=2494.98 tok/sec=94127.26 u/sec=24.21 ppl=7.526026 [2023-12-17:12:12:37:INFO:sockeye.training:__call__] E=17 B=71700 s/sec=2146.80 tok/sec=97402.23 u/sec=23.58 ppl=7.531691 [2023-12-17:12:12:39:INFO:sockeye.training:__call__] E=17 B=71750 s/sec=2092.12 tok/sec=96818.06 u/sec=23.27 ppl=7.539629 [2023-12-17:12:12:41:INFO:sockeye.training:__call__] E=17 B=71800 s/sec=2225.05 tok/sec=95592.66 u/sec=24.06 ppl=7.546406 [2023-12-17:12:12:44:INFO:sockeye.training:__call__] E=17 B=71850 s/sec=1935.58 tok/sec=101684.40 u/sec=23.09 ppl=7.550109 [2023-12-17:12:12:46:INFO:sockeye.training:__call__] E=17 B=71900 s/sec=1971.28 tok/sec=98903.21 u/sec=23.56 ppl=7.555808 [2023-12-17:12:12:48:INFO:sockeye.training:__call__] E=18 B=71950 s/sec=1799.28 tok/sec=96339.08 u/sec=21.58 ppl=7.546675 [2023-12-17:12:12:50:INFO:sockeye.training:__call__] E=18 B=72000 s/sec=1935.21 tok/sec=102446.24 u/sec=22.69 ppl=7.533156 [2023-12-17:12:12:50:INFO:sockeye.training:_create_checkpoint] Checkpoint [18] Updates=72000 Epoch=18 Samples=6416504 Time-cost=172.222 Updates/sec=23.226 [2023-12-17:12:12:50:INFO:sockeye.training:_create_checkpoint] Checkpoint [18] Train-ppl=7.533156 [2023-12-17:12:12:53:INFO:sockeye.training:_evaluate] Checkpoint [18] Validation-ppl=30.135231 Validation-bleu=0.177819 Validation-chrf=31.900580 Validation-rouge1=0.164104 Validation-rouge2=0.079882 Validation-rougel=0.155898 Validation-length-ratio-mse=1.277313 Validation-ter=95.130004 Validation-avg-sec-per-sent=0.003229 Validation-decode-walltime=1.614295 [2023-12-17:12:12:53:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 13 checkpoints, best so far: 31.983236 [2023-12-17:12:12:53:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [18] Learning-rate=0.000180 [2023-12-17:12:12:53:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:12:53:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=5, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:12:54:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00018" [2023-12-17:12:12:56:INFO:sockeye.training:__call__] E=18 B=72050 s/sec=780.72 tok/sec=40220.61 u/sec=8.79 ppl=6.655512 [2023-12-17:12:12:58:INFO:sockeye.training:__call__] E=18 B=72100 s/sec=1879.04 tok/sec=103578.41 u/sec=22.58 ppl=6.656816 [2023-12-17:12:13:00:INFO:sockeye.training:__call__] E=18 B=72150 s/sec=1704.86 tok/sec=102644.35 u/sec=22.96 ppl=6.608070 [2023-12-17:12:13:03:INFO:sockeye.training:__call__] E=18 B=72200 s/sec=2120.33 tok/sec=100523.82 u/sec=23.29 ppl=6.653016 [2023-12-17:12:13:05:INFO:sockeye.training:__call__] E=18 B=72250 s/sec=1914.67 tok/sec=102174.73 u/sec=23.06 ppl=6.663231 [2023-12-17:12:13:07:INFO:sockeye.training:__call__] E=18 B=72300 s/sec=1918.98 tok/sec=99075.74 u/sec=22.67 ppl=6.685006 [2023-12-17:12:13:09:INFO:sockeye.training:__call__] E=18 B=72350 s/sec=2173.09 tok/sec=98572.53 u/sec=23.10 ppl=6.709171 [2023-12-17:12:13:11:INFO:sockeye.training:__call__] E=18 B=72400 s/sec=2095.30 tok/sec=95789.51 u/sec=23.81 ppl=6.727787 [2023-12-17:12:13:13:INFO:sockeye.training:__call__] E=18 B=72450 s/sec=2057.83 tok/sec=97490.44 u/sec=23.91 ppl=6.745930 [2023-12-17:12:13:15:INFO:sockeye.training:__call__] E=18 B=72500 s/sec=2010.68 tok/sec=95446.20 u/sec=23.80 ppl=6.755170 [2023-12-17:12:13:18:INFO:sockeye.training:__call__] E=18 B=72550 s/sec=2098.03 tok/sec=101386.50 u/sec=23.13 ppl=6.767388 [2023-12-17:12:13:20:INFO:sockeye.training:__call__] E=18 B=72600 s/sec=1901.93 tok/sec=102902.32 u/sec=22.90 ppl=6.768543 [2023-12-17:12:13:22:INFO:sockeye.training:__call__] E=18 B=72650 s/sec=2229.19 tok/sec=95142.61 u/sec=23.94 ppl=6.781203 [2023-12-17:12:13:24:INFO:sockeye.training:__call__] E=18 B=72700 s/sec=2170.94 tok/sec=95653.68 u/sec=22.96 ppl=6.794539 [2023-12-17:12:13:26:INFO:sockeye.training:__call__] E=18 B=72750 s/sec=2093.71 tok/sec=98679.96 u/sec=23.28 ppl=6.802872 [2023-12-17:12:13:28:INFO:sockeye.training:__call__] E=18 B=72800 s/sec=2142.46 tok/sec=99961.69 u/sec=23.17 ppl=6.814816 [2023-12-17:12:13:30:INFO:sockeye.training:__call__] E=18 B=72850 s/sec=1982.16 tok/sec=99841.06 u/sec=23.37 ppl=6.822539 [2023-12-17:12:13:33:INFO:sockeye.training:__call__] E=18 B=72900 s/sec=2078.85 tok/sec=100954.11 u/sec=22.83 ppl=6.833322 [2023-12-17:12:13:35:INFO:sockeye.training:__call__] E=18 B=72950 s/sec=2140.11 tok/sec=98887.15 u/sec=23.42 ppl=6.841570 [2023-12-17:12:13:37:INFO:sockeye.training:__call__] E=18 B=73000 s/sec=1777.74 tok/sec=104865.30 u/sec=22.31 ppl=6.844534 [2023-12-17:12:13:39:INFO:sockeye.training:__call__] E=18 B=73050 s/sec=2151.17 tok/sec=98004.47 u/sec=23.18 ppl=6.861860 [2023-12-17:12:13:41:INFO:sockeye.training:__call__] E=18 B=73100 s/sec=1901.66 tok/sec=99218.28 u/sec=23.58 ppl=6.860430 [2023-12-17:12:13:43:INFO:sockeye.training:__call__] E=18 B=73150 s/sec=2102.72 tok/sec=96318.13 u/sec=23.55 ppl=6.867967 [2023-12-17:12:13:46:INFO:sockeye.training:__call__] E=18 B=73200 s/sec=2113.50 tok/sec=97571.60 u/sec=23.30 ppl=6.882972 [2023-12-17:12:13:48:INFO:sockeye.training:__call__] E=18 B=73250 s/sec=2353.46 tok/sec=94711.72 u/sec=23.57 ppl=6.901840 [2023-12-17:12:13:50:INFO:sockeye.training:__call__] E=18 B=73300 s/sec=2078.14 tok/sec=95955.85 u/sec=23.53 ppl=6.908409 [2023-12-17:12:13:52:INFO:sockeye.training:__call__] E=18 B=73350 s/sec=2239.86 tok/sec=96077.18 u/sec=23.89 ppl=6.926531 [2023-12-17:12:13:54:INFO:sockeye.training:__call__] E=18 B=73400 s/sec=2154.38 tok/sec=98498.60 u/sec=23.26 ppl=6.937321 [2023-12-17:12:13:56:INFO:sockeye.training:__call__] E=18 B=73450 s/sec=1973.21 tok/sec=101852.24 u/sec=22.67 ppl=6.947029 [2023-12-17:12:13:58:INFO:sockeye.training:__call__] E=18 B=73500 s/sec=2074.36 tok/sec=95130.57 u/sec=22.67 ppl=6.956728 [2023-12-17:12:14:01:INFO:sockeye.training:__call__] E=18 B=73550 s/sec=2007.64 tok/sec=100972.51 u/sec=22.77 ppl=6.961990 [2023-12-17:12:14:03:INFO:sockeye.training:__call__] E=18 B=73600 s/sec=2148.01 tok/sec=98062.30 u/sec=22.64 ppl=6.971984 [2023-12-17:12:14:05:INFO:sockeye.training:__call__] E=18 B=73650 s/sec=2129.30 tok/sec=100639.16 u/sec=23.10 ppl=6.980356 [2023-12-17:12:14:07:INFO:sockeye.training:__call__] E=18 B=73700 s/sec=2134.06 tok/sec=96409.52 u/sec=23.78 ppl=6.989750 [2023-12-17:12:14:09:INFO:sockeye.training:__call__] E=18 B=73750 s/sec=1972.13 tok/sec=99123.12 u/sec=23.39 ppl=6.997135 [2023-12-17:12:14:11:INFO:sockeye.training:__call__] E=18 B=73800 s/sec=2260.33 tok/sec=95819.74 u/sec=23.86 ppl=7.008366 [2023-12-17:12:14:14:INFO:sockeye.training:__call__] E=18 B=73850 s/sec=2006.85 tok/sec=108534.93 u/sec=21.48 ppl=7.017158 [2023-12-17:12:14:16:INFO:sockeye.training:__call__] E=18 B=73900 s/sec=2002.94 tok/sec=102322.11 u/sec=22.89 ppl=7.024320 [2023-12-17:12:14:18:INFO:sockeye.training:__call__] E=18 B=73950 s/sec=2016.18 tok/sec=102152.96 u/sec=23.08 ppl=7.031170 [2023-12-17:12:14:20:INFO:sockeye.training:__call__] E=18 B=74000 s/sec=1790.03 tok/sec=103988.95 u/sec=22.69 ppl=7.036245 [2023-12-17:12:14:22:INFO:sockeye.training:__call__] E=18 B=74050 s/sec=1906.08 tok/sec=103743.64 u/sec=22.95 ppl=7.043629 [2023-12-17:12:14:25:INFO:sockeye.training:__call__] E=18 B=74100 s/sec=1997.51 tok/sec=99778.95 u/sec=23.25 ppl=7.051516 [2023-12-17:12:14:27:INFO:sockeye.training:__call__] E=18 B=74150 s/sec=2022.70 tok/sec=101105.08 u/sec=23.24 ppl=7.056507 [2023-12-17:12:14:29:INFO:sockeye.training:__call__] E=18 B=74200 s/sec=1944.75 tok/sec=101918.18 u/sec=23.33 ppl=7.062079 [2023-12-17:12:14:31:INFO:sockeye.training:__call__] E=18 B=74250 s/sec=2282.48 tok/sec=95380.86 u/sec=23.98 ppl=7.072396 [2023-12-17:12:14:33:INFO:sockeye.training:__call__] E=18 B=74300 s/sec=2031.71 tok/sec=98357.33 u/sec=23.30 ppl=7.079931 [2023-12-17:12:14:35:INFO:sockeye.training:__call__] E=18 B=74350 s/sec=2085.85 tok/sec=98299.95 u/sec=23.32 ppl=7.089007 [2023-12-17:12:14:37:INFO:sockeye.training:__call__] E=18 B=74400 s/sec=2125.92 tok/sec=100147.96 u/sec=23.15 ppl=7.099747 [2023-12-17:12:14:40:INFO:sockeye.training:__call__] E=18 B=74450 s/sec=1981.12 tok/sec=101839.24 u/sec=23.32 ppl=7.103636 [2023-12-17:12:14:42:INFO:sockeye.training:__call__] E=18 B=74500 s/sec=2027.55 tok/sec=98588.85 u/sec=23.55 ppl=7.111374 [2023-12-17:12:14:44:INFO:sockeye.training:__call__] E=18 B=74550 s/sec=2069.18 tok/sec=99501.06 u/sec=23.47 ppl=7.118152 [2023-12-17:12:14:46:INFO:sockeye.training:__call__] E=18 B=74600 s/sec=2139.28 tok/sec=98399.24 u/sec=23.42 ppl=7.127467 [2023-12-17:12:14:48:INFO:sockeye.training:__call__] E=18 B=74650 s/sec=2049.83 tok/sec=98119.95 u/sec=23.59 ppl=7.135586 [2023-12-17:12:14:50:INFO:sockeye.training:__call__] E=18 B=74700 s/sec=2313.52 tok/sec=96321.72 u/sec=23.82 ppl=7.143660 [2023-12-17:12:14:52:INFO:sockeye.training:__call__] E=18 B=74750 s/sec=1977.21 tok/sec=102164.47 u/sec=22.59 ppl=7.149853 [2023-12-17:12:14:54:INFO:sockeye.training:__call__] E=18 B=74800 s/sec=2131.87 tok/sec=99277.90 u/sec=23.38 ppl=7.157881 [2023-12-17:12:14:57:INFO:sockeye.training:__call__] E=18 B=74850 s/sec=1994.77 tok/sec=97210.52 u/sec=22.83 ppl=7.165036 [2023-12-17:12:14:59:INFO:sockeye.training:__call__] E=18 B=74900 s/sec=2159.78 tok/sec=91995.00 u/sec=22.80 ppl=7.174223 [2023-12-17:12:15:01:INFO:sockeye.training:__call__] E=18 B=74950 s/sec=2075.39 tok/sec=97710.84 u/sec=23.29 ppl=7.183889 [2023-12-17:12:15:03:INFO:sockeye.training:__call__] E=18 B=75000 s/sec=1917.91 tok/sec=102551.33 u/sec=22.83 ppl=7.189054 [2023-12-17:12:15:05:INFO:sockeye.training:__call__] E=18 B=75050 s/sec=2233.38 tok/sec=97121.91 u/sec=23.58 ppl=7.195109 [2023-12-17:12:15:07:INFO:sockeye.training:__call__] E=18 B=75100 s/sec=2172.62 tok/sec=98432.84 u/sec=23.21 ppl=7.204277 [2023-12-17:12:15:10:INFO:sockeye.training:__call__] E=18 B=75150 s/sec=2177.74 tok/sec=96177.15 u/sec=23.23 ppl=7.213835 [2023-12-17:12:15:12:INFO:sockeye.training:__call__] E=18 B=75200 s/sec=1865.75 tok/sec=95712.49 u/sec=22.69 ppl=7.221176 [2023-12-17:12:15:14:INFO:sockeye.training:__call__] E=18 B=75250 s/sec=2284.32 tok/sec=93765.35 u/sec=23.37 ppl=7.232192 [2023-12-17:12:15:16:INFO:sockeye.training:__call__] E=18 B=75300 s/sec=2040.98 tok/sec=101670.06 u/sec=22.78 ppl=7.238938 [2023-12-17:12:15:18:INFO:sockeye.training:__call__] E=18 B=75350 s/sec=2213.97 tok/sec=98701.00 u/sec=23.37 ppl=7.245905 [2023-12-17:12:15:21:INFO:sockeye.training:__call__] E=18 B=75400 s/sec=1848.63 tok/sec=102622.75 u/sec=22.70 ppl=7.251069 [2023-12-17:12:15:23:INFO:sockeye.training:__call__] E=18 B=75450 s/sec=2292.96 tok/sec=95607.59 u/sec=23.92 ppl=7.258980 [2023-12-17:12:15:25:INFO:sockeye.training:__call__] E=18 B=75500 s/sec=2238.71 tok/sec=97953.08 u/sec=23.88 ppl=7.267207 [2023-12-17:12:15:27:INFO:sockeye.training:__call__] E=18 B=75550 s/sec=2157.97 tok/sec=98603.79 u/sec=23.02 ppl=7.276090 [2023-12-17:12:15:29:INFO:sockeye.training:__call__] E=18 B=75600 s/sec=1970.74 tok/sec=97986.78 u/sec=23.24 ppl=7.280593 [2023-12-17:12:15:31:INFO:sockeye.training:__call__] E=18 B=75650 s/sec=2042.35 tok/sec=96133.14 u/sec=22.83 ppl=7.287321 [2023-12-17:12:15:33:INFO:sockeye.training:__call__] E=18 B=75700 s/sec=1957.03 tok/sec=103421.90 u/sec=22.12 ppl=7.291122 [2023-12-17:12:15:36:INFO:sockeye.training:__call__] E=18 B=75750 s/sec=2199.21 tok/sec=96233.28 u/sec=23.74 ppl=7.297548 [2023-12-17:12:15:38:INFO:sockeye.training:__call__] E=18 B=75800 s/sec=2136.73 tok/sec=100210.54 u/sec=23.27 ppl=7.305508 [2023-12-17:12:15:40:INFO:sockeye.training:__call__] E=18 B=75850 s/sec=2156.63 tok/sec=98622.53 u/sec=23.04 ppl=7.313422 [2023-12-17:12:15:42:INFO:sockeye.training:__call__] E=18 B=75900 s/sec=2008.72 tok/sec=98977.84 u/sec=23.25 ppl=7.318254 [2023-12-17:12:15:44:INFO:sockeye.training:__call__] E=19 B=75950 s/sec=1854.60 tok/sec=94955.67 u/sec=21.87 ppl=7.309117 [2023-12-17:12:15:47:INFO:sockeye.training:__call__] E=19 B=76000 s/sec=1855.74 tok/sec=101712.20 u/sec=23.01 ppl=7.296937 [2023-12-17:12:15:47:INFO:sockeye.training:_create_checkpoint] Checkpoint [19] Updates=76000 Epoch=19 Samples=6772576 Time-cost=172.779 Updates/sec=23.151 [2023-12-17:12:15:47:INFO:sockeye.training:_create_checkpoint] Checkpoint [19] Train-ppl=7.296937 [2023-12-17:12:15:49:INFO:sockeye.training:_evaluate] Checkpoint [19] Validation-ppl=31.853540 Validation-bleu=0.161435 Validation-chrf=31.492669 Validation-rouge1=0.167680 Validation-rouge2=0.080657 Validation-rougel=0.159907 Validation-length-ratio-mse=1.288595 Validation-ter=95.749071 Validation-avg-sec-per-sent=0.003272 Validation-decode-walltime=1.636203 [2023-12-17:12:15:49:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 14 checkpoints, best so far: 31.983236 [2023-12-17:12:15:49:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [19] Learning-rate=0.000180 [2023-12-17:12:15:50:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:15:50:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=6, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:15:50:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00019" [2023-12-17:12:15:52:INFO:sockeye.training:__call__] E=19 B=76050 s/sec=871.22 tok/sec=36990.87 u/sec=8.85 ppl=6.569601 [2023-12-17:12:15:54:INFO:sockeye.training:__call__] E=19 B=76100 s/sec=2155.34 tok/sec=92877.63 u/sec=23.35 ppl=6.550563 [2023-12-17:12:15:56:INFO:sockeye.training:__call__] E=19 B=76150 s/sec=2220.98 tok/sec=98654.36 u/sec=23.10 ppl=6.521896 [2023-12-17:12:15:59:INFO:sockeye.training:__call__] E=19 B=76200 s/sec=2056.74 tok/sec=99375.64 u/sec=23.16 ppl=6.532642 [2023-12-17:12:16:01:INFO:sockeye.training:__call__] E=19 B=76250 s/sec=2053.36 tok/sec=99845.38 u/sec=23.33 ppl=6.537674 [2023-12-17:12:16:03:INFO:sockeye.training:__call__] E=19 B=76300 s/sec=2070.28 tok/sec=97573.37 u/sec=23.44 ppl=6.537692 [2023-12-17:12:16:05:INFO:sockeye.training:__call__] E=19 B=76350 s/sec=1953.09 tok/sec=97669.70 u/sec=23.93 ppl=6.548775 [2023-12-17:12:16:07:INFO:sockeye.training:__call__] E=19 B=76400 s/sec=1787.06 tok/sec=101581.61 u/sec=23.08 ppl=6.548746 [2023-12-17:12:16:09:INFO:sockeye.training:__call__] E=19 B=76450 s/sec=2180.11 tok/sec=97226.07 u/sec=23.37 ppl=6.565093 [2023-12-17:12:16:12:INFO:sockeye.training:__call__] E=19 B=76500 s/sec=1921.11 tok/sec=102895.79 u/sec=22.74 ppl=6.571258 [2023-12-17:12:16:14:INFO:sockeye.training:__call__] E=19 B=76550 s/sec=2424.18 tok/sec=93416.23 u/sec=23.71 ppl=6.597521 [2023-12-17:12:16:16:INFO:sockeye.training:__call__] E=19 B=76600 s/sec=2179.76 tok/sec=98306.35 u/sec=23.41 ppl=6.609164 [2023-12-17:12:16:18:INFO:sockeye.training:__call__] E=19 B=76650 s/sec=2116.19 tok/sec=95140.94 u/sec=23.83 ppl=6.622841 [2023-12-17:12:16:20:INFO:sockeye.training:__call__] E=19 B=76700 s/sec=2237.88 tok/sec=97890.05 u/sec=23.71 ppl=6.632985 [2023-12-17:12:16:22:INFO:sockeye.training:__call__] E=19 B=76750 s/sec=2180.30 tok/sec=96472.65 u/sec=23.41 ppl=6.648442 [2023-12-17:12:16:24:INFO:sockeye.training:__call__] E=19 B=76800 s/sec=1904.20 tok/sec=99957.23 u/sec=23.29 ppl=6.655141 [2023-12-17:12:16:26:INFO:sockeye.training:__call__] E=19 B=76850 s/sec=2088.76 tok/sec=97147.99 u/sec=23.61 ppl=6.664548 [2023-12-17:12:16:28:INFO:sockeye.training:__call__] E=19 B=76900 s/sec=2209.52 tok/sec=97142.97 u/sec=23.77 ppl=6.677757 [2023-12-17:12:16:31:INFO:sockeye.training:__call__] E=19 B=76950 s/sec=1838.94 tok/sec=101278.75 u/sec=22.54 ppl=6.681769 [2023-12-17:12:16:33:INFO:sockeye.training:__call__] E=19 B=77000 s/sec=2216.78 tok/sec=98745.36 u/sec=23.29 ppl=6.688867 [2023-12-17:12:16:35:INFO:sockeye.training:__call__] E=19 B=77050 s/sec=1909.70 tok/sec=101927.74 u/sec=22.52 ppl=6.690539 [2023-12-17:12:16:37:INFO:sockeye.training:__call__] E=19 B=77100 s/sec=2076.57 tok/sec=95514.65 u/sec=23.38 ppl=6.703369 [2023-12-17:12:16:39:INFO:sockeye.training:__call__] E=19 B=77150 s/sec=1968.03 tok/sec=100881.78 u/sec=23.21 ppl=6.707748 [2023-12-17:12:16:41:INFO:sockeye.training:__call__] E=19 B=77200 s/sec=2138.25 tok/sec=94422.85 u/sec=24.12 ppl=6.716942 [2023-12-17:12:16:44:INFO:sockeye.training:__call__] E=19 B=77250 s/sec=2095.33 tok/sec=97522.66 u/sec=23.39 ppl=6.725604 [2023-12-17:12:16:46:INFO:sockeye.training:__call__] E=19 B=77300 s/sec=2187.98 tok/sec=97899.94 u/sec=23.78 ppl=6.736841 [2023-12-17:12:16:48:INFO:sockeye.training:__call__] E=19 B=77350 s/sec=2183.08 tok/sec=98366.02 u/sec=23.44 ppl=6.747660 [2023-12-17:12:16:50:INFO:sockeye.training:__call__] E=19 B=77400 s/sec=2033.18 tok/sec=103622.46 u/sec=22.98 ppl=6.751646 [2023-12-17:12:16:52:INFO:sockeye.training:__call__] E=19 B=77450 s/sec=2089.82 tok/sec=99494.96 u/sec=23.62 ppl=6.762566 [2023-12-17:12:16:54:INFO:sockeye.training:__call__] E=19 B=77500 s/sec=2101.17 tok/sec=100111.11 u/sec=23.28 ppl=6.772397 [2023-12-17:12:16:56:INFO:sockeye.training:__call__] E=19 B=77550 s/sec=2040.25 tok/sec=101815.57 u/sec=22.81 ppl=6.779693 [2023-12-17:12:16:59:INFO:sockeye.training:__call__] E=19 B=77600 s/sec=2151.17 tok/sec=98467.34 u/sec=23.38 ppl=6.787984 [2023-12-17:12:17:01:INFO:sockeye.training:__call__] E=19 B=77650 s/sec=2030.52 tok/sec=101206.90 u/sec=23.20 ppl=6.795412 [2023-12-17:12:17:03:INFO:sockeye.training:__call__] E=19 B=77700 s/sec=2163.39 tok/sec=98668.62 u/sec=23.64 ppl=6.803980 [2023-12-17:12:17:05:INFO:sockeye.training:__call__] E=19 B=77750 s/sec=1868.70 tok/sec=101674.91 u/sec=22.77 ppl=6.808807 [2023-12-17:12:17:07:INFO:sockeye.training:__call__] E=19 B=77800 s/sec=1992.46 tok/sec=97465.86 u/sec=23.41 ppl=6.815867 [2023-12-17:12:17:09:INFO:sockeye.training:__call__] E=19 B=77850 s/sec=2113.34 tok/sec=96832.09 u/sec=23.84 ppl=6.825154 [2023-12-17:12:17:11:INFO:sockeye.training:__call__] E=19 B=77900 s/sec=2143.75 tok/sec=97681.59 u/sec=23.84 ppl=6.833995 [2023-12-17:12:17:13:INFO:sockeye.training:__call__] E=19 B=77950 s/sec=2193.94 tok/sec=97752.74 u/sec=23.52 ppl=6.844472 [2023-12-17:12:17:16:INFO:sockeye.training:__call__] E=19 B=78000 s/sec=1916.81 tok/sec=102820.87 u/sec=23.08 ppl=6.851425 [2023-12-17:12:17:18:INFO:sockeye.training:__call__] E=19 B=78050 s/sec=1996.10 tok/sec=103840.69 u/sec=22.56 ppl=6.857741 [2023-12-17:12:17:20:INFO:sockeye.training:__call__] E=19 B=78100 s/sec=1957.84 tok/sec=101350.43 u/sec=23.04 ppl=6.866029 [2023-12-17:12:17:22:INFO:sockeye.training:__call__] E=19 B=78150 s/sec=2005.24 tok/sec=98875.67 u/sec=23.30 ppl=6.872853 [2023-12-17:12:17:24:INFO:sockeye.training:__call__] E=19 B=78200 s/sec=2111.57 tok/sec=99162.33 u/sec=23.15 ppl=6.880788 [2023-12-17:12:17:26:INFO:sockeye.training:__call__] E=19 B=78250 s/sec=2040.49 tok/sec=98213.94 u/sec=23.49 ppl=6.889473 [2023-12-17:12:17:29:INFO:sockeye.training:__call__] E=19 B=78300 s/sec=1779.57 tok/sec=104524.82 u/sec=22.24 ppl=6.895321 [2023-12-17:12:17:31:INFO:sockeye.training:__call__] E=19 B=78350 s/sec=2191.09 tok/sec=99269.77 u/sec=23.29 ppl=6.905513 [2023-12-17:12:17:33:INFO:sockeye.training:__call__] E=19 B=78400 s/sec=2004.07 tok/sec=101707.46 u/sec=23.15 ppl=6.910663 [2023-12-17:12:17:35:INFO:sockeye.training:__call__] E=19 B=78450 s/sec=2184.55 tok/sec=97244.40 u/sec=23.42 ppl=6.919477 [2023-12-17:12:17:37:INFO:sockeye.training:__call__] E=19 B=78500 s/sec=2121.43 tok/sec=98664.93 u/sec=23.10 ppl=6.927117 [2023-12-17:12:17:39:INFO:sockeye.training:__call__] E=19 B=78550 s/sec=2099.57 tok/sec=97673.29 u/sec=23.56 ppl=6.933558 [2023-12-17:12:17:42:INFO:sockeye.training:__call__] E=19 B=78600 s/sec=2076.05 tok/sec=97484.26 u/sec=23.51 ppl=6.942225 [2023-12-17:12:17:44:INFO:sockeye.training:__call__] E=19 B=78650 s/sec=1936.15 tok/sec=101048.46 u/sec=23.05 ppl=6.945034 [2023-12-17:12:17:46:INFO:sockeye.training:__call__] E=19 B=78700 s/sec=2195.18 tok/sec=94070.38 u/sec=23.99 ppl=6.953214 [2023-12-17:12:17:48:INFO:sockeye.training:__call__] E=19 B=78750 s/sec=2218.54 tok/sec=98853.90 u/sec=23.03 ppl=6.962781 [2023-12-17:12:17:50:INFO:sockeye.training:__call__] E=19 B=78800 s/sec=2186.40 tok/sec=97203.23 u/sec=23.36 ppl=6.972182 [2023-12-17:12:17:52:INFO:sockeye.training:__call__] E=19 B=78850 s/sec=2039.03 tok/sec=101387.81 u/sec=22.88 ppl=6.979170 [2023-12-17:12:17:54:INFO:sockeye.training:__call__] E=19 B=78900 s/sec=2205.08 tok/sec=99448.21 u/sec=23.24 ppl=6.987292 [2023-12-17:12:17:57:INFO:sockeye.training:__call__] E=19 B=78950 s/sec=1909.42 tok/sec=101086.93 u/sec=23.40 ppl=6.995012 [2023-12-17:12:17:59:INFO:sockeye.training:__call__] E=19 B=79000 s/sec=2369.11 tok/sec=93851.59 u/sec=23.77 ppl=7.004389 [2023-12-17:12:18:01:INFO:sockeye.training:__call__] E=19 B=79050 s/sec=1999.80 tok/sec=101710.71 u/sec=23.28 ppl=7.010500 [2023-12-17:12:18:03:INFO:sockeye.training:__call__] E=19 B=79100 s/sec=2076.70 tok/sec=101428.82 u/sec=23.14 ppl=7.018921 [2023-12-17:12:18:05:INFO:sockeye.training:__call__] E=19 B=79150 s/sec=1921.84 tok/sec=101616.78 u/sec=22.45 ppl=7.024153 [2023-12-17:12:18:07:INFO:sockeye.training:__call__] E=19 B=79200 s/sec=2032.98 tok/sec=100293.90 u/sec=23.02 ppl=7.030531 [2023-12-17:12:18:10:INFO:sockeye.training:__call__] E=19 B=79250 s/sec=1938.14 tok/sec=103855.76 u/sec=22.43 ppl=7.034799 [2023-12-17:12:18:12:INFO:sockeye.training:__call__] E=19 B=79300 s/sec=2181.14 tok/sec=104893.21 u/sec=22.13 ppl=7.041060 [2023-12-17:12:18:14:INFO:sockeye.training:__call__] E=19 B=79350 s/sec=2028.57 tok/sec=99626.34 u/sec=23.18 ppl=7.048179 [2023-12-17:12:18:16:INFO:sockeye.training:__call__] E=19 B=79400 s/sec=1915.09 tok/sec=101954.93 u/sec=22.58 ppl=7.052150 [2023-12-17:12:18:19:INFO:sockeye.training:__call__] E=19 B=79450 s/sec=1934.92 tok/sec=103486.64 u/sec=21.99 ppl=7.056580 [2023-12-17:12:18:21:INFO:sockeye.training:__call__] E=19 B=79500 s/sec=1922.59 tok/sec=103602.18 u/sec=22.67 ppl=7.061821 [2023-12-17:12:18:23:INFO:sockeye.training:__call__] E=19 B=79550 s/sec=2200.67 tok/sec=99090.44 u/sec=23.55 ppl=7.069725 [2023-12-17:12:18:25:INFO:sockeye.training:__call__] E=19 B=79600 s/sec=1918.02 tok/sec=105433.19 u/sec=22.70 ppl=7.074300 [2023-12-17:12:18:27:INFO:sockeye.training:__call__] E=19 B=79650 s/sec=2087.01 tok/sec=100652.75 u/sec=23.25 ppl=7.078458 [2023-12-17:12:18:29:INFO:sockeye.training:__call__] E=19 B=79700 s/sec=2059.35 tok/sec=99299.01 u/sec=23.44 ppl=7.083422 [2023-12-17:12:18:31:INFO:sockeye.training:__call__] E=19 B=79750 s/sec=2290.75 tok/sec=95880.21 u/sec=24.06 ppl=7.091527 [2023-12-17:12:18:34:INFO:sockeye.training:__call__] E=19 B=79800 s/sec=2262.48 tok/sec=94716.23 u/sec=24.05 ppl=7.097999 [2023-12-17:12:18:36:INFO:sockeye.training:__call__] E=19 B=79850 s/sec=2221.61 tok/sec=99032.99 u/sec=23.30 ppl=7.106331 [2023-12-17:12:18:38:INFO:sockeye.training:__call__] E=19 B=79900 s/sec=1875.05 tok/sec=102103.15 u/sec=23.11 ppl=7.113401 [2023-12-17:12:18:40:INFO:sockeye.training:__call__] E=20 B=79950 s/sec=1857.21 tok/sec=97921.68 u/sec=21.70 ppl=7.101489 [2023-12-17:12:18:42:INFO:sockeye.training:__call__] E=20 B=80000 s/sec=2243.26 tok/sec=97391.29 u/sec=23.56 ppl=7.092047 [2023-12-17:12:18:42:INFO:sockeye.training:_create_checkpoint] Checkpoint [20] Updates=80000 Epoch=20 Samples=7129840 Time-cost=172.315 Updates/sec=23.213 [2023-12-17:12:18:42:INFO:sockeye.training:_create_checkpoint] Checkpoint [20] Train-ppl=7.092047 [2023-12-17:12:18:45:INFO:sockeye.training:_evaluate] Checkpoint [20] Validation-ppl=32.059560 Validation-bleu=0.158444 Validation-chrf=30.095456 Validation-rouge1=0.156664 Validation-rouge2=0.074008 Validation-rougel=0.149906 Validation-length-ratio-mse=1.275953 Validation-ter=95.955427 Validation-avg-sec-per-sent=0.003679 Validation-decode-walltime=1.839712 [2023-12-17:12:18:45:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 15 checkpoints, best so far: 31.983236 [2023-12-17:12:18:45:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 31.983236 > 0.000000 over 20 checkpoints [2023-12-17:12:18:45:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [20] Learning-rate=0.000180 [2023-12-17:12:18:46:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:18:46:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=7, base_lr=0.0002, lr=0.00018, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:18:46:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00020" [2023-12-17:12:18:48:INFO:sockeye.training:__call__] E=20 B=80050 s/sec=760.85 tok/sec=38022.91 u/sec=8.76 ppl=6.285661 [2023-12-17:12:18:50:INFO:sockeye.training:__call__] E=20 B=80100 s/sec=1933.72 tok/sec=100612.24 u/sec=22.98 ppl=6.313898 [2023-12-17:12:18:52:INFO:sockeye.training:__call__] E=20 B=80150 s/sec=2122.89 tok/sec=96459.42 u/sec=23.61 ppl=6.343372 [2023-12-17:12:18:54:INFO:sockeye.training:__call__] E=20 B=80200 s/sec=2361.44 tok/sec=97031.03 u/sec=23.65 ppl=6.376526 [2023-12-17:12:18:57:INFO:sockeye.training:__call__] E=20 B=80250 s/sec=1997.03 tok/sec=100751.22 u/sec=22.49 ppl=6.379919 [2023-12-17:12:18:59:INFO:sockeye.training:__call__] E=20 B=80300 s/sec=1982.44 tok/sec=100016.99 u/sec=22.73 ppl=6.396636 [2023-12-17:12:19:01:INFO:sockeye.training:__call__] E=20 B=80350 s/sec=1775.81 tok/sec=103177.84 u/sec=22.60 ppl=6.395059 [2023-12-17:12:19:03:INFO:sockeye.training:__call__] E=20 B=80400 s/sec=2162.08 tok/sec=96493.73 u/sec=23.92 ppl=6.413126 [2023-12-17:12:19:05:INFO:sockeye.training:__call__] E=20 B=80450 s/sec=2150.84 tok/sec=99103.73 u/sec=23.42 ppl=6.426313 [2023-12-17:12:19:07:INFO:sockeye.training:__call__] E=20 B=80500 s/sec=2246.97 tok/sec=98313.15 u/sec=23.37 ppl=6.439665 [2023-12-17:12:19:10:INFO:sockeye.training:__call__] E=20 B=80550 s/sec=1986.08 tok/sec=99379.45 u/sec=23.60 ppl=6.449165 [2023-12-17:12:19:12:INFO:sockeye.training:__call__] E=20 B=80600 s/sec=2207.10 tok/sec=99040.67 u/sec=23.54 ppl=6.464504 [2023-12-17:12:19:14:INFO:sockeye.training:__call__] E=20 B=80650 s/sec=1948.32 tok/sec=100041.29 u/sec=23.37 ppl=6.472466 [2023-12-17:12:19:16:INFO:sockeye.training:__call__] E=20 B=80700 s/sec=2064.52 tok/sec=100438.35 u/sec=23.29 ppl=6.479078 [2023-12-17:12:19:18:INFO:sockeye.training:__call__] E=20 B=80750 s/sec=2056.12 tok/sec=101493.83 u/sec=22.79 ppl=6.481815 [2023-12-17:12:19:20:INFO:sockeye.training:__call__] E=20 B=80800 s/sec=1968.67 tok/sec=100350.31 u/sec=23.22 ppl=6.484483 [2023-12-17:12:19:22:INFO:sockeye.training:__call__] E=20 B=80850 s/sec=2035.10 tok/sec=97203.23 u/sec=23.51 ppl=6.494877 [2023-12-17:12:19:25:INFO:sockeye.training:__call__] E=20 B=80900 s/sec=2253.11 tok/sec=97352.44 u/sec=23.63 ppl=6.513594 [2023-12-17:12:19:27:INFO:sockeye.training:__call__] E=20 B=80950 s/sec=2176.81 tok/sec=98020.22 u/sec=23.50 ppl=6.521518 [2023-12-17:12:19:29:INFO:sockeye.training:__call__] E=20 B=81000 s/sec=2078.98 tok/sec=98862.20 u/sec=23.41 ppl=6.530746 [2023-12-17:12:19:31:INFO:sockeye.training:__call__] E=20 B=81050 s/sec=2170.44 tok/sec=97741.41 u/sec=23.51 ppl=6.541063 [2023-12-17:12:19:33:INFO:sockeye.training:__call__] E=20 B=81100 s/sec=2189.11 tok/sec=96701.48 u/sec=23.79 ppl=6.555166 [2023-12-17:12:19:35:INFO:sockeye.training:__call__] E=20 B=81150 s/sec=2139.92 tok/sec=101859.98 u/sec=23.06 ppl=6.563903 [2023-12-17:12:19:37:INFO:sockeye.training:__call__] E=20 B=81200 s/sec=2094.53 tok/sec=99758.18 u/sec=23.42 ppl=6.569619 [2023-12-17:12:19:40:INFO:sockeye.training:__call__] E=20 B=81250 s/sec=2191.60 tok/sec=97054.33 u/sec=23.22 ppl=6.582074 [2023-12-17:12:19:42:INFO:sockeye.training:__call__] E=20 B=81300 s/sec=2184.32 tok/sec=98242.03 u/sec=23.46 ppl=6.591335 [2023-12-17:12:19:44:INFO:sockeye.training:__call__] E=20 B=81350 s/sec=2297.76 tok/sec=95191.63 u/sec=23.74 ppl=6.607169 [2023-12-17:12:19:46:INFO:sockeye.training:__call__] E=20 B=81400 s/sec=2050.95 tok/sec=98713.77 u/sec=23.26 ppl=6.613617 [2023-12-17:12:19:48:INFO:sockeye.training:__call__] E=20 B=81450 s/sec=2037.34 tok/sec=97914.08 u/sec=23.76 ppl=6.620028 [2023-12-17:12:19:50:INFO:sockeye.training:__call__] E=20 B=81500 s/sec=2127.56 tok/sec=102210.54 u/sec=22.89 ppl=6.628505 [2023-12-17:12:19:52:INFO:sockeye.training:__call__] E=20 B=81550 s/sec=1932.03 tok/sec=99147.54 u/sec=23.40 ppl=6.634301 [2023-12-17:12:19:55:INFO:sockeye.training:__call__] E=20 B=81600 s/sec=2038.30 tok/sec=101708.32 u/sec=23.08 ppl=6.639507 [2023-12-17:12:19:57:INFO:sockeye.training:__call__] E=20 B=81650 s/sec=1758.89 tok/sec=103587.82 u/sec=22.85 ppl=6.647408 [2023-12-17:12:19:59:INFO:sockeye.training:__call__] E=20 B=81700 s/sec=2143.49 tok/sec=96826.39 u/sec=23.80 ppl=6.657153 [2023-12-17:12:20:01:INFO:sockeye.training:__call__] E=20 B=81750 s/sec=2020.64 tok/sec=103188.07 u/sec=22.84 ppl=6.661757 [2023-12-17:12:20:03:INFO:sockeye.training:__call__] E=20 B=81800 s/sec=2259.08 tok/sec=94873.82 u/sec=24.14 ppl=6.673292 [2023-12-17:12:20:05:INFO:sockeye.training:__call__] E=20 B=81850 s/sec=2259.39 tok/sec=94105.11 u/sec=23.50 ppl=6.682556 [2023-12-17:12:20:07:INFO:sockeye.training:__call__] E=20 B=81900 s/sec=2096.56 tok/sec=104538.02 u/sec=22.10 ppl=6.690091 [2023-12-17:12:20:10:INFO:sockeye.training:__call__] E=20 B=81950 s/sec=2214.91 tok/sec=96968.95 u/sec=23.79 ppl=6.697361 [2023-12-17:12:20:12:INFO:sockeye.training:__call__] E=20 B=82000 s/sec=2000.30 tok/sec=98231.87 u/sec=23.41 ppl=6.705483 [2023-12-17:12:20:14:INFO:sockeye.training:__call__] E=20 B=82050 s/sec=1843.68 tok/sec=100827.44 u/sec=23.05 ppl=6.711568 [2023-12-17:12:20:16:INFO:sockeye.training:__call__] E=20 B=82100 s/sec=2028.68 tok/sec=101075.79 u/sec=23.31 ppl=6.719622 [2023-12-17:12:20:18:INFO:sockeye.training:__call__] E=20 B=82150 s/sec=2117.88 tok/sec=100251.47 u/sec=23.39 ppl=6.727101 [2023-12-17:12:20:20:INFO:sockeye.training:__call__] E=20 B=82200 s/sec=2047.89 tok/sec=98388.78 u/sec=23.48 ppl=6.733320 [2023-12-17:12:20:22:INFO:sockeye.training:__call__] E=20 B=82250 s/sec=2040.51 tok/sec=99541.33 u/sec=23.10 ppl=6.741435 [2023-12-17:12:20:25:INFO:sockeye.training:__call__] E=20 B=82300 s/sec=1961.49 tok/sec=101214.52 u/sec=23.09 ppl=6.748946 [2023-12-17:12:20:27:INFO:sockeye.training:__call__] E=20 B=82350 s/sec=2117.04 tok/sec=99245.09 u/sec=23.17 ppl=6.758100 [2023-12-17:12:20:29:INFO:sockeye.training:__call__] E=20 B=82400 s/sec=2282.65 tok/sec=95579.43 u/sec=24.02 ppl=6.767465 [2023-12-17:12:20:31:INFO:sockeye.training:__call__] E=20 B=82450 s/sec=2060.14 tok/sec=99898.20 u/sec=23.24 ppl=6.773712 [2023-12-17:12:20:33:INFO:sockeye.training:__call__] E=20 B=82500 s/sec=2017.66 tok/sec=100428.34 u/sec=22.93 ppl=6.778262 [2023-12-17:12:20:35:INFO:sockeye.training:__call__] E=20 B=82550 s/sec=2114.71 tok/sec=97451.71 u/sec=23.81 ppl=6.783685 [2023-12-17:12:20:37:INFO:sockeye.training:__call__] E=20 B=82600 s/sec=2055.61 tok/sec=101217.70 u/sec=23.15 ppl=6.789914 [2023-12-17:12:20:40:INFO:sockeye.training:__call__] E=20 B=82650 s/sec=1865.72 tok/sec=103165.97 u/sec=22.82 ppl=6.795774 [2023-12-17:12:20:42:INFO:sockeye.training:__call__] E=20 B=82700 s/sec=1969.50 tok/sec=104379.61 u/sec=22.75 ppl=6.802006 [2023-12-17:12:20:44:INFO:sockeye.training:__call__] E=20 B=82750 s/sec=2140.74 tok/sec=97738.08 u/sec=23.47 ppl=6.809593 [2023-12-17:12:20:46:INFO:sockeye.training:__call__] E=20 B=82800 s/sec=2043.28 tok/sec=100624.16 u/sec=23.13 ppl=6.813652 [2023-12-17:12:20:48:INFO:sockeye.training:__call__] E=20 B=82850 s/sec=2005.40 tok/sec=102699.62 u/sec=22.66 ppl=6.818660 [2023-12-17:12:20:50:INFO:sockeye.training:__call__] E=20 B=82900 s/sec=2059.45 tok/sec=97330.73 u/sec=23.79 ppl=6.824484 [2023-12-17:12:20:53:INFO:sockeye.training:__call__] E=20 B=82950 s/sec=2032.26 tok/sec=101714.14 u/sec=22.60 ppl=6.831115 [2023-12-17:12:20:55:INFO:sockeye.training:__call__] E=20 B=83000 s/sec=2161.86 tok/sec=97342.71 u/sec=23.14 ppl=6.838728 [2023-12-17:12:20:57:INFO:sockeye.training:__call__] E=20 B=83050 s/sec=2347.81 tok/sec=97892.85 u/sec=23.04 ppl=6.846215 [2023-12-17:12:20:59:INFO:sockeye.training:__call__] E=20 B=83100 s/sec=2079.68 tok/sec=95543.05 u/sec=23.89 ppl=6.852022 [2023-12-17:12:21:01:INFO:sockeye.training:__call__] E=20 B=83150 s/sec=2075.99 tok/sec=98129.65 u/sec=23.25 ppl=6.858401 [2023-12-17:12:21:03:INFO:sockeye.training:__call__] E=20 B=83200 s/sec=2180.53 tok/sec=96337.12 u/sec=23.66 ppl=6.865254 [2023-12-17:12:21:06:INFO:sockeye.training:__call__] E=20 B=83250 s/sec=1984.92 tok/sec=99791.85 u/sec=23.06 ppl=6.871397 [2023-12-17:12:21:08:INFO:sockeye.training:__call__] E=20 B=83300 s/sec=1902.28 tok/sec=103443.56 u/sec=22.73 ppl=6.873577 [2023-12-17:12:21:10:INFO:sockeye.training:__call__] E=20 B=83350 s/sec=2125.88 tok/sec=101895.82 u/sec=22.91 ppl=6.879540 [2023-12-17:12:21:12:INFO:sockeye.training:__call__] E=20 B=83400 s/sec=2043.16 tok/sec=99047.59 u/sec=23.82 ppl=6.885876 [2023-12-17:12:21:14:INFO:sockeye.training:__call__] E=20 B=83450 s/sec=2154.41 tok/sec=98968.62 u/sec=23.30 ppl=6.891840 [2023-12-17:12:21:16:INFO:sockeye.training:__call__] E=20 B=83500 s/sec=1854.33 tok/sec=101262.57 u/sec=22.90 ppl=6.897275 [2023-12-17:12:21:18:INFO:sockeye.training:__call__] E=20 B=83550 s/sec=2131.91 tok/sec=95950.96 u/sec=23.62 ppl=6.904540 [2023-12-17:12:21:21:INFO:sockeye.training:__call__] E=20 B=83600 s/sec=2014.12 tok/sec=100326.46 u/sec=22.80 ppl=6.910046 [2023-12-17:12:21:23:INFO:sockeye.training:__call__] E=20 B=83650 s/sec=2069.05 tok/sec=98346.28 u/sec=23.64 ppl=6.915024 [2023-12-17:12:21:25:INFO:sockeye.training:__call__] E=20 B=83700 s/sec=1970.74 tok/sec=102257.63 u/sec=23.02 ppl=6.919572 [2023-12-17:12:21:27:INFO:sockeye.training:__call__] E=20 B=83750 s/sec=2166.13 tok/sec=98484.69 u/sec=23.18 ppl=6.926217 [2023-12-17:12:21:29:INFO:sockeye.training:__call__] E=20 B=83800 s/sec=1876.30 tok/sec=102449.44 u/sec=22.77 ppl=6.931338 [2023-12-17:12:21:31:INFO:sockeye.training:__call__] E=20 B=83850 s/sec=1950.69 tok/sec=103706.79 u/sec=22.75 ppl=6.935958 [2023-12-17:12:21:34:INFO:sockeye.training:__call__] E=21 B=83900 s/sec=1883.50 tok/sec=95198.52 u/sec=21.76 ppl=6.940595 [2023-12-17:12:21:36:INFO:sockeye.training:__call__] E=21 B=83950 s/sec=2144.56 tok/sec=97507.03 u/sec=23.19 ppl=6.931458 [2023-12-17:12:21:38:INFO:sockeye.training:__call__] E=21 B=84000 s/sec=1968.90 tok/sec=102672.59 u/sec=22.62 ppl=6.922625 [2023-12-17:12:21:38:INFO:sockeye.training:_create_checkpoint] Checkpoint [21] Updates=84000 Epoch=21 Samples=7486136 Time-cost=172.291 Updates/sec=23.216 [2023-12-17:12:21:38:INFO:sockeye.training:_create_checkpoint] Checkpoint [21] Train-ppl=6.922625 [2023-12-17:12:21:41:INFO:sockeye.training:_evaluate] Checkpoint [21] Validation-ppl=32.811089 Validation-bleu=0.149172 Validation-chrf=28.994237 Validation-rouge1=0.152784 Validation-rouge2=0.074439 Validation-rougel=0.145765 Validation-length-ratio-mse=1.345232 Validation-ter=100.247627 Validation-avg-sec-per-sent=0.003303 Validation-decode-walltime=1.651370 [2023-12-17:12:21:41:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 16 checkpoints, best so far: 31.983236 [2023-12-17:12:21:41:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 3.215049 > 0.000000 over 20 checkpoints [2023-12-17:12:21:41:INFO:sockeye.lr_scheduler:new_evaluation_result] 8 checkpoints since improvement or rate scaling, lowering learning rate: 1.80e-04 -> 1.62e-04 [2023-12-17:12:21:41:INFO:sockeye.training:_adjust_learning_rate] Loading model parameters and optimizer states from best checkpoint: 5 [2023-12-17:12:21:41:INFO:sockeye.model:load_parameters] Loaded params from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best" to "cuda:0" [2023-12-17:12:21:41:INFO:sockeye.training:_load_optimizer_state] Loaded optimizer state from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl" [2023-12-17:12:21:41:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [21] Learning-rate=0.000180 [2023-12-17:12:21:42:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:21:42:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:21:42:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00021" [2023-12-17:12:21:44:INFO:sockeye.training:__call__] E=21 B=84050 s/sec=728.75 tok/sec=35500.36 u/sec=8.30 ppl=8.235107 [2023-12-17:12:21:46:INFO:sockeye.training:__call__] E=21 B=84100 s/sec=2229.26 tok/sec=97271.49 u/sec=23.62 ppl=8.250975 [2023-12-17:12:21:48:INFO:sockeye.training:__call__] E=21 B=84150 s/sec=2106.77 tok/sec=100279.42 u/sec=22.78 ppl=8.255132 [2023-12-17:12:21:51:INFO:sockeye.training:__call__] E=21 B=84200 s/sec=2086.60 tok/sec=99467.32 u/sec=23.41 ppl=8.263311 [2023-12-17:12:21:53:INFO:sockeye.training:__call__] E=21 B=84250 s/sec=2077.08 tok/sec=98981.19 u/sec=23.39 ppl=8.273792 [2023-12-17:12:21:55:INFO:sockeye.training:__call__] E=21 B=84300 s/sec=1935.33 tok/sec=102056.82 u/sec=22.87 ppl=8.252656 [2023-12-17:12:21:57:INFO:sockeye.training:__call__] E=21 B=84350 s/sec=2112.87 tok/sec=101506.80 u/sec=23.25 ppl=8.263791 [2023-12-17:12:21:59:INFO:sockeye.training:__call__] E=21 B=84400 s/sec=2162.21 tok/sec=97182.16 u/sec=23.67 ppl=8.292101 [2023-12-17:12:22:01:INFO:sockeye.training:__call__] E=21 B=84450 s/sec=1734.71 tok/sec=104991.52 u/sec=22.54 ppl=8.272024 [2023-12-17:12:22:04:INFO:sockeye.training:__call__] E=21 B=84500 s/sec=2061.76 tok/sec=98781.67 u/sec=23.82 ppl=8.280892 [2023-12-17:12:22:06:INFO:sockeye.training:__call__] E=21 B=84550 s/sec=2342.67 tok/sec=96697.99 u/sec=23.85 ppl=8.300305 [2023-12-17:12:22:08:INFO:sockeye.training:__call__] E=21 B=84600 s/sec=2045.07 tok/sec=100850.66 u/sec=23.07 ppl=8.304572 [2023-12-17:12:22:10:INFO:sockeye.training:__call__] E=21 B=84650 s/sec=2101.95 tok/sec=97806.45 u/sec=23.42 ppl=8.306284 [2023-12-17:12:22:12:INFO:sockeye.training:__call__] E=21 B=84700 s/sec=2104.69 tok/sec=97197.01 u/sec=23.36 ppl=8.311434 [2023-12-17:12:22:14:INFO:sockeye.training:__call__] E=21 B=84750 s/sec=2152.57 tok/sec=97110.76 u/sec=23.56 ppl=8.331409 [2023-12-17:12:22:16:INFO:sockeye.training:__call__] E=21 B=84800 s/sec=1963.11 tok/sec=99857.11 u/sec=23.02 ppl=8.337479 [2023-12-17:12:22:18:INFO:sockeye.training:__call__] E=21 B=84850 s/sec=2135.63 tok/sec=97448.12 u/sec=23.54 ppl=8.356623 [2023-12-17:12:22:21:INFO:sockeye.training:__call__] E=21 B=84900 s/sec=2080.20 tok/sec=97493.97 u/sec=23.90 ppl=8.376492 [2023-12-17:12:22:23:INFO:sockeye.training:__call__] E=21 B=84950 s/sec=1902.13 tok/sec=101664.21 u/sec=23.13 ppl=8.382380 [2023-12-17:12:22:25:INFO:sockeye.training:__call__] E=21 B=85000 s/sec=2092.42 tok/sec=99206.06 u/sec=23.44 ppl=8.386780 [2023-12-17:12:22:27:INFO:sockeye.training:__call__] E=21 B=85050 s/sec=1924.81 tok/sec=102203.58 u/sec=23.09 ppl=8.386099 [2023-12-17:12:22:29:INFO:sockeye.training:__call__] E=21 B=85100 s/sec=1874.97 tok/sec=104006.38 u/sec=22.80 ppl=8.388428 [2023-12-17:12:22:31:INFO:sockeye.training:__call__] E=21 B=85150 s/sec=2272.69 tok/sec=95993.59 u/sec=23.79 ppl=8.400283 [2023-12-17:12:22:33:INFO:sockeye.training:__call__] E=21 B=85200 s/sec=1901.15 tok/sec=101364.81 u/sec=23.16 ppl=8.405705 [2023-12-17:12:22:36:INFO:sockeye.training:__call__] E=21 B=85250 s/sec=2112.28 tok/sec=98487.34 u/sec=23.62 ppl=8.410861 [2023-12-17:12:22:38:INFO:sockeye.training:__call__] E=21 B=85300 s/sec=1976.16 tok/sec=98843.98 u/sec=22.58 ppl=8.421157 [2023-12-17:12:22:40:INFO:sockeye.training:__call__] E=21 B=85350 s/sec=1928.50 tok/sec=103244.58 u/sec=22.91 ppl=8.426752 [2023-12-17:12:22:42:INFO:sockeye.training:__call__] E=21 B=85400 s/sec=2142.95 tok/sec=96904.96 u/sec=23.62 ppl=8.442187 [2023-12-17:12:22:44:INFO:sockeye.training:__call__] E=21 B=85450 s/sec=2308.88 tok/sec=98031.84 u/sec=23.46 ppl=8.457267 [2023-12-17:12:22:46:INFO:sockeye.training:__call__] E=21 B=85500 s/sec=1954.11 tok/sec=100120.93 u/sec=22.87 ppl=8.469481 [2023-12-17:12:22:49:INFO:sockeye.training:__call__] E=21 B=85550 s/sec=1752.07 tok/sec=113921.19 u/sec=20.86 ppl=8.472761 [2023-12-17:12:22:51:INFO:sockeye.training:__call__] E=21 B=85600 s/sec=2153.62 tok/sec=98769.52 u/sec=23.21 ppl=8.480860 [2023-12-17:12:22:53:INFO:sockeye.training:__call__] E=21 B=85650 s/sec=2411.78 tok/sec=96592.41 u/sec=23.66 ppl=8.497657 [2023-12-17:12:22:55:INFO:sockeye.training:__call__] E=21 B=85700 s/sec=2230.54 tok/sec=100047.13 u/sec=23.23 ppl=8.518990 [2023-12-17:12:22:57:INFO:sockeye.training:__call__] E=21 B=85750 s/sec=2318.70 tok/sec=95728.20 u/sec=23.76 ppl=8.535432 [2023-12-17:12:22:59:INFO:sockeye.training:__call__] E=21 B=85800 s/sec=2099.77 tok/sec=98389.31 u/sec=23.43 ppl=8.546487 [2023-12-17:12:23:02:INFO:sockeye.training:__call__] E=21 B=85850 s/sec=1954.07 tok/sec=100911.55 u/sec=22.79 ppl=8.551546 [2023-12-17:12:23:04:INFO:sockeye.training:__call__] E=21 B=85900 s/sec=2019.44 tok/sec=101960.13 u/sec=22.70 ppl=8.559439 [2023-12-17:12:23:06:INFO:sockeye.training:__call__] E=21 B=85950 s/sec=2293.59 tok/sec=97515.41 u/sec=23.73 ppl=8.567747 [2023-12-17:12:23:08:INFO:sockeye.training:__call__] E=21 B=86000 s/sec=2163.87 tok/sec=96843.55 u/sec=23.85 ppl=8.575915 [2023-12-17:12:23:10:INFO:sockeye.training:__call__] E=21 B=86050 s/sec=2168.57 tok/sec=98010.22 u/sec=23.29 ppl=8.585305 [2023-12-17:12:23:12:INFO:sockeye.training:__call__] E=21 B=86100 s/sec=2209.45 tok/sec=97860.99 u/sec=23.44 ppl=8.596834 [2023-12-17:12:23:14:INFO:sockeye.training:__call__] E=21 B=86150 s/sec=2288.49 tok/sec=98114.67 u/sec=23.56 ppl=8.607284 [2023-12-17:12:23:17:INFO:sockeye.training:__call__] E=21 B=86200 s/sec=2096.51 tok/sec=98562.01 u/sec=23.44 ppl=8.618154 [2023-12-17:12:23:19:INFO:sockeye.training:__call__] E=21 B=86250 s/sec=2139.05 tok/sec=98911.60 u/sec=23.33 ppl=8.628411 [2023-12-17:12:23:21:INFO:sockeye.training:__call__] E=21 B=86300 s/sec=1977.63 tok/sec=100393.75 u/sec=22.72 ppl=8.632874 [2023-12-17:12:23:23:INFO:sockeye.training:__call__] E=21 B=86350 s/sec=1949.89 tok/sec=100254.50 u/sec=23.12 ppl=8.637703 [2023-12-17:12:23:25:INFO:sockeye.training:__call__] E=21 B=86400 s/sec=1990.72 tok/sec=100468.39 u/sec=23.13 ppl=8.639429 [2023-12-17:12:23:27:INFO:sockeye.training:__call__] E=21 B=86450 s/sec=2113.70 tok/sec=97237.87 u/sec=23.72 ppl=8.651461 [2023-12-17:12:23:30:INFO:sockeye.training:__call__] E=21 B=86500 s/sec=1829.98 tok/sec=101226.54 u/sec=23.29 ppl=8.656353 [2023-12-17:12:23:32:INFO:sockeye.training:__call__] E=21 B=86550 s/sec=2268.07 tok/sec=94627.96 u/sec=23.47 ppl=8.667628 [2023-12-17:12:23:34:INFO:sockeye.training:__call__] E=21 B=86600 s/sec=2140.13 tok/sec=97188.80 u/sec=23.38 ppl=8.676264 [2023-12-17:12:23:36:INFO:sockeye.training:__call__] E=21 B=86650 s/sec=2101.69 tok/sec=100350.36 u/sec=23.04 ppl=8.684607 [2023-12-17:12:23:38:INFO:sockeye.training:__call__] E=21 B=86700 s/sec=2327.68 tok/sec=95701.35 u/sec=24.13 ppl=8.699639 [2023-12-17:12:23:40:INFO:sockeye.training:__call__] E=21 B=86750 s/sec=2424.98 tok/sec=93137.66 u/sec=23.76 ppl=8.708366 [2023-12-17:12:23:42:INFO:sockeye.training:__call__] E=21 B=86800 s/sec=2222.33 tok/sec=97377.91 u/sec=23.42 ppl=8.716655 [2023-12-17:12:23:44:INFO:sockeye.training:__call__] E=21 B=86850 s/sec=2100.33 tok/sec=98158.18 u/sec=23.07 ppl=8.723593 [2023-12-17:12:23:47:INFO:sockeye.training:__call__] E=21 B=86900 s/sec=1976.44 tok/sec=99932.29 u/sec=23.13 ppl=8.729459 [2023-12-17:12:23:49:INFO:sockeye.training:__call__] E=21 B=86950 s/sec=1830.71 tok/sec=102316.68 u/sec=22.66 ppl=8.735782 [2023-12-17:12:23:51:INFO:sockeye.training:__call__] E=21 B=87000 s/sec=2146.02 tok/sec=94905.30 u/sec=23.45 ppl=8.742850 [2023-12-17:12:23:53:INFO:sockeye.training:__call__] E=21 B=87050 s/sec=2036.21 tok/sec=97947.90 u/sec=23.39 ppl=8.749377 [2023-12-17:12:23:55:INFO:sockeye.training:__call__] E=21 B=87100 s/sec=1923.42 tok/sec=102777.95 u/sec=22.94 ppl=8.755489 [2023-12-17:12:23:57:INFO:sockeye.training:__call__] E=21 B=87150 s/sec=2147.56 tok/sec=97245.16 u/sec=23.63 ppl=8.760519 [2023-12-17:12:24:00:INFO:sockeye.training:__call__] E=21 B=87200 s/sec=1935.62 tok/sec=100338.93 u/sec=23.31 ppl=8.765941 [2023-12-17:12:24:02:INFO:sockeye.training:__call__] E=21 B=87250 s/sec=1764.73 tok/sec=102982.46 u/sec=22.56 ppl=8.769765 [2023-12-17:12:24:04:INFO:sockeye.training:__call__] E=21 B=87300 s/sec=2072.62 tok/sec=100197.81 u/sec=23.17 ppl=8.775374 [2023-12-17:12:24:06:INFO:sockeye.training:__call__] E=21 B=87350 s/sec=2144.82 tok/sec=98252.61 u/sec=23.68 ppl=8.782972 [2023-12-17:12:24:08:INFO:sockeye.training:__call__] E=21 B=87400 s/sec=2055.07 tok/sec=101432.39 u/sec=22.94 ppl=8.789146 [2023-12-17:12:24:10:INFO:sockeye.training:__call__] E=21 B=87450 s/sec=1990.22 tok/sec=99570.07 u/sec=23.03 ppl=8.794666 [2023-12-17:12:24:13:INFO:sockeye.training:__call__] E=21 B=87500 s/sec=2037.96 tok/sec=99143.86 u/sec=22.95 ppl=8.803141 [2023-12-17:12:24:15:INFO:sockeye.training:__call__] E=21 B=87550 s/sec=2072.22 tok/sec=97541.14 u/sec=23.51 ppl=8.807621 [2023-12-17:12:24:17:INFO:sockeye.training:__call__] E=21 B=87600 s/sec=2226.58 tok/sec=96022.91 u/sec=23.95 ppl=8.814576 [2023-12-17:12:24:19:INFO:sockeye.training:__call__] E=21 B=87650 s/sec=1910.89 tok/sec=99752.75 u/sec=23.24 ppl=8.821069 [2023-12-17:12:24:21:INFO:sockeye.training:__call__] E=21 B=87700 s/sec=2080.84 tok/sec=98689.33 u/sec=23.27 ppl=8.828689 [2023-12-17:12:24:23:INFO:sockeye.training:__call__] E=21 B=87750 s/sec=2157.19 tok/sec=99870.57 u/sec=23.53 ppl=8.837337 [2023-12-17:12:24:25:INFO:sockeye.training:__call__] E=21 B=87800 s/sec=1985.96 tok/sec=103298.62 u/sec=22.57 ppl=8.845083 [2023-12-17:12:24:28:INFO:sockeye.training:__call__] E=21 B=87850 s/sec=1757.68 tok/sec=103699.52 u/sec=22.74 ppl=8.845849 [2023-12-17:12:24:30:INFO:sockeye.training:__call__] E=22 B=87900 s/sec=1887.97 tok/sec=96641.51 u/sec=22.18 ppl=8.847503 [2023-12-17:12:24:32:INFO:sockeye.training:__call__] E=22 B=87950 s/sec=1851.62 tok/sec=104470.30 u/sec=22.56 ppl=8.827910 [2023-12-17:12:24:34:INFO:sockeye.training:__call__] E=22 B=88000 s/sec=2004.53 tok/sec=98240.54 u/sec=23.24 ppl=8.811137 [2023-12-17:12:24:34:INFO:sockeye.training:_create_checkpoint] Checkpoint [22] Updates=88000 Epoch=22 Samples=7842232 Time-cost=172.242 Updates/sec=23.223 [2023-12-17:12:24:34:INFO:sockeye.training:_create_checkpoint] Checkpoint [22] Train-ppl=8.811137 [2023-12-17:12:24:37:INFO:sockeye.training:_evaluate] Checkpoint [22] Validation-ppl=27.261654 Validation-bleu=0.153659 Validation-chrf=30.401315 Validation-rouge1=0.164726 Validation-rouge2=0.077968 Validation-rougel=0.158489 Validation-length-ratio-mse=1.258889 Validation-ter=95.253818 Validation-avg-sec-per-sent=0.003145 Validation-decode-walltime=1.572691 [2023-12-17:12:24:37:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 17 checkpoints, best so far: 31.983236 [2023-12-17:12:24:37:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 1.383042 > 0.000000 over 20 checkpoints [2023-12-17:12:24:37:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [22] Learning-rate=0.000162 [2023-12-17:12:24:37:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:24:37:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:24:38:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00022" [2023-12-17:12:24:40:INFO:sockeye.training:__call__] E=22 B=88050 s/sec=850.44 tok/sec=36089.55 u/sec=8.99 ppl=7.690584 [2023-12-17:12:24:42:INFO:sockeye.training:__call__] E=22 B=88100 s/sec=2042.30 tok/sec=99409.59 u/sec=23.42 ppl=7.743049 [2023-12-17:12:24:44:INFO:sockeye.training:__call__] E=22 B=88150 s/sec=2087.54 tok/sec=99963.59 u/sec=23.26 ppl=7.687191 [2023-12-17:12:24:46:INFO:sockeye.training:__call__] E=22 B=88200 s/sec=2155.93 tok/sec=97655.63 u/sec=23.35 ppl=7.681806 [2023-12-17:12:24:48:INFO:sockeye.training:__call__] E=22 B=88250 s/sec=1798.29 tok/sec=100233.37 u/sec=22.98 ppl=7.666700 [2023-12-17:12:24:51:INFO:sockeye.training:__call__] E=22 B=88300 s/sec=2082.08 tok/sec=98042.26 u/sec=23.53 ppl=7.698421 [2023-12-17:12:24:53:INFO:sockeye.training:__call__] E=22 B=88350 s/sec=2045.59 tok/sec=100560.11 u/sec=23.16 ppl=7.704035 [2023-12-17:12:24:55:INFO:sockeye.training:__call__] E=22 B=88400 s/sec=1812.00 tok/sec=103039.98 u/sec=23.11 ppl=7.720610 [2023-12-17:12:24:57:INFO:sockeye.training:__call__] E=22 B=88450 s/sec=1786.02 tok/sec=105091.94 u/sec=22.15 ppl=7.739639 [2023-12-17:12:24:59:INFO:sockeye.training:__call__] E=22 B=88500 s/sec=1993.93 tok/sec=101122.34 u/sec=22.62 ppl=7.764221 [2023-12-17:12:25:01:INFO:sockeye.training:__call__] E=22 B=88550 s/sec=1944.88 tok/sec=99746.64 u/sec=23.56 ppl=7.771814 [2023-12-17:12:25:04:INFO:sockeye.training:__call__] E=22 B=88600 s/sec=2058.19 tok/sec=97956.24 u/sec=23.35 ppl=7.778841 [2023-12-17:12:25:06:INFO:sockeye.training:__call__] E=22 B=88650 s/sec=1991.03 tok/sec=101013.85 u/sec=22.63 ppl=7.785477 [2023-12-17:12:25:08:INFO:sockeye.training:__call__] E=22 B=88700 s/sec=2173.83 tok/sec=101743.72 u/sec=22.57 ppl=7.796485 [2023-12-17:12:25:10:INFO:sockeye.training:__call__] E=22 B=88750 s/sec=2029.52 tok/sec=99420.54 u/sec=23.02 ppl=7.822881 [2023-12-17:12:25:12:INFO:sockeye.training:__call__] E=22 B=88800 s/sec=2026.37 tok/sec=97205.38 u/sec=23.50 ppl=7.839229 [2023-12-17:12:25:14:INFO:sockeye.training:__call__] E=22 B=88850 s/sec=2124.02 tok/sec=98096.39 u/sec=23.54 ppl=7.855832 [2023-12-17:12:25:17:INFO:sockeye.training:__call__] E=22 B=88900 s/sec=1925.65 tok/sec=101427.31 u/sec=22.97 ppl=7.867652 [2023-12-17:12:25:19:INFO:sockeye.training:__call__] E=22 B=88950 s/sec=2078.53 tok/sec=96793.33 u/sec=23.97 ppl=7.881715 [2023-12-17:12:25:21:INFO:sockeye.training:__call__] E=22 B=89000 s/sec=2161.70 tok/sec=99468.01 u/sec=23.29 ppl=7.899899 [2023-12-17:12:25:23:INFO:sockeye.training:__call__] E=22 B=89050 s/sec=2086.55 tok/sec=97985.11 u/sec=23.54 ppl=7.914262 [2023-12-17:12:25:25:INFO:sockeye.training:__call__] E=22 B=89100 s/sec=1966.89 tok/sec=101713.00 u/sec=22.64 ppl=7.920670 [2023-12-17:12:25:27:INFO:sockeye.training:__call__] E=22 B=89150 s/sec=2153.95 tok/sec=100522.73 u/sec=23.09 ppl=7.930490 [2023-12-17:12:25:30:INFO:sockeye.training:__call__] E=22 B=89200 s/sec=1821.35 tok/sec=102314.94 u/sec=22.68 ppl=7.943069 [2023-12-17:12:25:32:INFO:sockeye.training:__call__] E=22 B=89250 s/sec=2194.23 tok/sec=96864.34 u/sec=23.69 ppl=7.952993 [2023-12-17:12:25:34:INFO:sockeye.training:__call__] E=22 B=89300 s/sec=1973.18 tok/sec=101604.08 u/sec=23.01 ppl=7.962889 [2023-12-17:12:25:36:INFO:sockeye.training:__call__] E=22 B=89350 s/sec=2002.07 tok/sec=99740.61 u/sec=23.13 ppl=7.973974 [2023-12-17:12:25:38:INFO:sockeye.training:__call__] E=22 B=89400 s/sec=2105.90 tok/sec=99051.87 u/sec=23.34 ppl=7.979236 [2023-12-17:12:25:40:INFO:sockeye.training:__call__] E=22 B=89450 s/sec=2052.68 tok/sec=100621.31 u/sec=22.71 ppl=7.985352 [2023-12-17:12:25:43:INFO:sockeye.training:__call__] E=22 B=89500 s/sec=1839.24 tok/sec=102880.43 u/sec=22.85 ppl=7.985802 [2023-12-17:12:25:45:INFO:sockeye.training:__call__] E=22 B=89550 s/sec=2187.75 tok/sec=98460.15 u/sec=23.37 ppl=7.997873 [2023-12-17:12:25:47:INFO:sockeye.training:__call__] E=22 B=89600 s/sec=2266.63 tok/sec=94577.04 u/sec=23.69 ppl=8.010507 [2023-12-17:12:25:49:INFO:sockeye.training:__call__] E=22 B=89650 s/sec=1981.10 tok/sec=101414.58 u/sec=22.97 ppl=8.019476 [2023-12-17:12:25:51:INFO:sockeye.training:__call__] E=22 B=89700 s/sec=2217.68 tok/sec=96663.51 u/sec=23.82 ppl=8.026947 [2023-12-17:12:25:53:INFO:sockeye.training:__call__] E=22 B=89750 s/sec=2348.38 tok/sec=95505.98 u/sec=23.60 ppl=8.039519 [2023-12-17:12:25:55:INFO:sockeye.training:__call__] E=22 B=89800 s/sec=1900.19 tok/sec=101778.93 u/sec=23.20 ppl=8.040463 [2023-12-17:12:25:58:INFO:sockeye.training:__call__] E=22 B=89850 s/sec=2054.62 tok/sec=99865.78 u/sec=23.14 ppl=8.050833 [2023-12-17:12:26:00:INFO:sockeye.training:__call__] E=22 B=89900 s/sec=2248.63 tok/sec=96959.56 u/sec=23.62 ppl=8.064299 [2023-12-17:12:26:02:INFO:sockeye.training:__call__] E=22 B=89950 s/sec=2273.17 tok/sec=94951.04 u/sec=23.64 ppl=8.076224 [2023-12-17:12:26:04:INFO:sockeye.training:__call__] E=22 B=90000 s/sec=1768.09 tok/sec=102079.46 u/sec=22.60 ppl=8.078272 [2023-12-17:12:26:06:INFO:sockeye.training:__call__] E=22 B=90050 s/sec=2398.51 tok/sec=94021.44 u/sec=24.18 ppl=8.093981 [2023-12-17:12:26:08:INFO:sockeye.training:__call__] E=22 B=90100 s/sec=2050.15 tok/sec=97438.80 u/sec=23.64 ppl=8.104998 [2023-12-17:12:26:10:INFO:sockeye.training:__call__] E=22 B=90150 s/sec=2089.14 tok/sec=98783.91 u/sec=23.36 ppl=8.111726 [2023-12-17:12:26:12:INFO:sockeye.training:__call__] E=22 B=90200 s/sec=2214.68 tok/sec=98383.18 u/sec=23.62 ppl=8.124278 [2023-12-17:12:26:15:INFO:sockeye.training:__call__] E=22 B=90250 s/sec=2104.66 tok/sec=101960.91 u/sec=22.88 ppl=8.132759 [2023-12-17:12:26:17:INFO:sockeye.training:__call__] E=22 B=90300 s/sec=2001.76 tok/sec=100562.66 u/sec=23.17 ppl=8.139170 [2023-12-17:12:26:19:INFO:sockeye.training:__call__] E=22 B=90350 s/sec=1750.13 tok/sec=104223.36 u/sec=22.69 ppl=8.143618 [2023-12-17:12:26:21:INFO:sockeye.training:__call__] E=22 B=90400 s/sec=2028.14 tok/sec=100894.41 u/sec=22.88 ppl=8.148310 [2023-12-17:12:26:23:INFO:sockeye.training:__call__] E=22 B=90450 s/sec=1944.57 tok/sec=99656.28 u/sec=23.06 ppl=8.152051 [2023-12-17:12:26:25:INFO:sockeye.training:__call__] E=22 B=90500 s/sec=2125.82 tok/sec=97824.37 u/sec=23.03 ppl=8.160944 [2023-12-17:12:26:28:INFO:sockeye.training:__call__] E=22 B=90550 s/sec=2282.78 tok/sec=94324.30 u/sec=24.02 ppl=8.170304 [2023-12-17:12:26:30:INFO:sockeye.training:__call__] E=22 B=90600 s/sec=1762.48 tok/sec=101703.22 u/sec=23.24 ppl=8.179457 [2023-12-17:12:26:32:INFO:sockeye.training:__call__] E=22 B=90650 s/sec=2122.88 tok/sec=98453.40 u/sec=23.61 ppl=8.185735 [2023-12-17:12:26:34:INFO:sockeye.training:__call__] E=22 B=90700 s/sec=2019.15 tok/sec=100443.33 u/sec=22.94 ppl=8.194206 [2023-12-17:12:26:36:INFO:sockeye.training:__call__] E=22 B=90750 s/sec=2080.42 tok/sec=102854.40 u/sec=22.93 ppl=8.202190 [2023-12-17:12:26:38:INFO:sockeye.training:__call__] E=22 B=90800 s/sec=2326.28 tok/sec=97065.19 u/sec=23.76 ppl=8.213530 [2023-12-17:12:26:40:INFO:sockeye.training:__call__] E=22 B=90850 s/sec=1982.13 tok/sec=101040.30 u/sec=23.16 ppl=8.219073 [2023-12-17:12:26:43:INFO:sockeye.training:__call__] E=22 B=90900 s/sec=2300.97 tok/sec=97749.66 u/sec=23.73 ppl=8.229367 [2023-12-17:12:26:45:INFO:sockeye.training:__call__] E=22 B=90950 s/sec=1891.99 tok/sec=101483.98 u/sec=22.96 ppl=8.235350 [2023-12-17:12:26:47:INFO:sockeye.training:__call__] E=22 B=91000 s/sec=2045.69 tok/sec=97983.08 u/sec=23.42 ppl=8.241601 [2023-12-17:12:26:49:INFO:sockeye.training:__call__] E=22 B=91050 s/sec=2001.77 tok/sec=101091.23 u/sec=22.87 ppl=8.249103 [2023-12-17:12:26:51:INFO:sockeye.training:__call__] E=22 B=91100 s/sec=2071.79 tok/sec=99178.84 u/sec=23.16 ppl=8.252113 [2023-12-17:12:26:53:INFO:sockeye.training:__call__] E=22 B=91150 s/sec=2166.13 tok/sec=97835.08 u/sec=23.63 ppl=8.259324 [2023-12-17:12:26:56:INFO:sockeye.training:__call__] E=22 B=91200 s/sec=1972.43 tok/sec=103598.11 u/sec=22.70 ppl=8.265624 [2023-12-17:12:26:58:INFO:sockeye.training:__call__] E=22 B=91250 s/sec=2266.94 tok/sec=96337.53 u/sec=23.77 ppl=8.272195 [2023-12-17:12:27:00:INFO:sockeye.training:__call__] E=22 B=91300 s/sec=2131.33 tok/sec=99110.64 u/sec=23.54 ppl=8.279450 [2023-12-17:12:27:02:INFO:sockeye.training:__call__] E=22 B=91350 s/sec=1960.52 tok/sec=107943.98 u/sec=21.84 ppl=8.286165 [2023-12-17:12:27:04:INFO:sockeye.training:__call__] E=22 B=91400 s/sec=1925.30 tok/sec=100916.55 u/sec=22.75 ppl=8.292078 [2023-12-17:12:27:06:INFO:sockeye.training:__call__] E=22 B=91450 s/sec=2262.15 tok/sec=96223.48 u/sec=23.84 ppl=8.300497 [2023-12-17:12:27:09:INFO:sockeye.training:__call__] E=22 B=91500 s/sec=2040.50 tok/sec=99991.85 u/sec=23.19 ppl=8.307018 [2023-12-17:12:27:11:INFO:sockeye.training:__call__] E=22 B=91550 s/sec=2298.85 tok/sec=97238.63 u/sec=23.59 ppl=8.317038 [2023-12-17:12:27:13:INFO:sockeye.training:__call__] E=22 B=91600 s/sec=2141.72 tok/sec=98458.83 u/sec=23.57 ppl=8.323484 [2023-12-17:12:27:15:INFO:sockeye.training:__call__] E=22 B=91650 s/sec=2215.70 tok/sec=99811.99 u/sec=23.55 ppl=8.331444 [2023-12-17:12:27:17:INFO:sockeye.training:__call__] E=22 B=91700 s/sec=2301.97 tok/sec=93096.68 u/sec=24.10 ppl=8.340017 [2023-12-17:12:27:19:INFO:sockeye.training:__call__] E=22 B=91750 s/sec=2145.29 tok/sec=97894.50 u/sec=23.48 ppl=8.348316 [2023-12-17:12:27:21:INFO:sockeye.training:__call__] E=22 B=91800 s/sec=2362.02 tok/sec=95174.97 u/sec=23.58 ppl=8.358826 [2023-12-17:12:27:23:INFO:sockeye.training:__call__] E=22 B=91850 s/sec=2154.66 tok/sec=99219.49 u/sec=23.46 ppl=8.364689 [2023-12-17:12:27:26:INFO:sockeye.training:__call__] E=23 B=91900 s/sec=1918.96 tok/sec=93450.09 u/sec=21.81 ppl=8.363632 [2023-12-17:12:27:28:INFO:sockeye.training:__call__] E=23 B=91950 s/sec=2142.69 tok/sec=96637.35 u/sec=23.33 ppl=8.348428 [2023-12-17:12:27:30:INFO:sockeye.training:__call__] E=23 B=92000 s/sec=2321.70 tok/sec=96352.26 u/sec=23.67 ppl=8.331606 [2023-12-17:12:27:30:INFO:sockeye.training:_create_checkpoint] Checkpoint [23] Updates=92000 Epoch=23 Samples=8199752 Time-cost=172.169 Updates/sec=23.233 [2023-12-17:12:27:30:INFO:sockeye.training:_create_checkpoint] Checkpoint [23] Train-ppl=8.331606 [2023-12-17:12:27:33:INFO:sockeye.training:_evaluate] Checkpoint [23] Validation-ppl=28.039788 Validation-bleu=0.180385 Validation-chrf=31.658328 Validation-rouge1=0.164596 Validation-rouge2=0.080206 Validation-rougel=0.157144 Validation-length-ratio-mse=1.245799 Validation-ter=94.841106 Validation-avg-sec-per-sent=0.003411 Validation-decode-walltime=1.705528 [2023-12-17:12:27:33:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 18 checkpoints, best so far: 31.983236 [2023-12-17:12:27:33:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 1.383042 > 0.000000 over 20 checkpoints [2023-12-17:12:27:33:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [23] Learning-rate=0.000162 [2023-12-17:12:27:33:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:27:33:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:27:33:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00023" [2023-12-17:12:27:35:INFO:sockeye.training:__call__] E=23 B=92050 s/sec=881.95 tok/sec=37595.56 u/sec=8.92 ppl=7.386309 [2023-12-17:12:27:38:INFO:sockeye.training:__call__] E=23 B=92100 s/sec=1924.83 tok/sec=99070.21 u/sec=22.78 ppl=7.214556 [2023-12-17:12:27:40:INFO:sockeye.training:__call__] E=23 B=92150 s/sec=2042.54 tok/sec=100346.44 u/sec=23.38 ppl=7.233553 [2023-12-17:12:27:42:INFO:sockeye.training:__call__] E=23 B=92200 s/sec=2067.43 tok/sec=99087.86 u/sec=23.28 ppl=7.270839 [2023-12-17:12:27:44:INFO:sockeye.training:__call__] E=23 B=92250 s/sec=2183.30 tok/sec=95040.33 u/sec=23.90 ppl=7.307340 [2023-12-17:12:27:46:INFO:sockeye.training:__call__] E=23 B=92300 s/sec=2140.69 tok/sec=95548.86 u/sec=23.85 ppl=7.346610 [2023-12-17:12:27:48:INFO:sockeye.training:__call__] E=23 B=92350 s/sec=2157.69 tok/sec=100781.07 u/sec=23.17 ppl=7.381501 [2023-12-17:12:27:50:INFO:sockeye.training:__call__] E=23 B=92400 s/sec=2187.89 tok/sec=98612.75 u/sec=23.50 ppl=7.390512 [2023-12-17:12:27:53:INFO:sockeye.training:__call__] E=23 B=92450 s/sec=1779.73 tok/sec=103320.38 u/sec=23.03 ppl=7.381797 [2023-12-17:12:27:55:INFO:sockeye.training:__call__] E=23 B=92500 s/sec=1950.14 tok/sec=100946.73 u/sec=23.22 ppl=7.376257 [2023-12-17:12:27:57:INFO:sockeye.training:__call__] E=23 B=92550 s/sec=2148.85 tok/sec=100447.65 u/sec=23.16 ppl=7.387144 [2023-12-17:12:27:59:INFO:sockeye.training:__call__] E=23 B=92600 s/sec=2005.06 tok/sec=99870.72 u/sec=23.42 ppl=7.403480 [2023-12-17:12:28:01:INFO:sockeye.training:__call__] E=23 B=92650 s/sec=2103.58 tok/sec=100680.37 u/sec=22.79 ppl=7.414194 [2023-12-17:12:28:03:INFO:sockeye.training:__call__] E=23 B=92700 s/sec=1908.30 tok/sec=103656.87 u/sec=22.38 ppl=7.410044 [2023-12-17:12:28:06:INFO:sockeye.training:__call__] E=23 B=92750 s/sec=1836.52 tok/sec=102142.49 u/sec=22.86 ppl=7.415441 [2023-12-17:12:28:08:INFO:sockeye.training:__call__] E=23 B=92800 s/sec=2089.70 tok/sec=97968.94 u/sec=23.41 ppl=7.422983 [2023-12-17:12:28:10:INFO:sockeye.training:__call__] E=23 B=92850 s/sec=2051.21 tok/sec=99287.51 u/sec=23.14 ppl=7.440219 [2023-12-17:12:28:12:INFO:sockeye.training:__call__] E=23 B=92900 s/sec=1667.56 tok/sec=104850.35 u/sec=22.17 ppl=7.450023 [2023-12-17:12:28:14:INFO:sockeye.training:__call__] E=23 B=92950 s/sec=2111.63 tok/sec=98396.76 u/sec=23.61 ppl=7.462761 [2023-12-17:12:28:16:INFO:sockeye.training:__call__] E=23 B=93000 s/sec=2339.54 tok/sec=95477.53 u/sec=23.89 ppl=7.479897 [2023-12-17:12:28:19:INFO:sockeye.training:__call__] E=23 B=93050 s/sec=2251.31 tok/sec=99249.12 u/sec=23.03 ppl=7.492569 [2023-12-17:12:28:21:INFO:sockeye.training:__call__] E=23 B=93100 s/sec=1801.72 tok/sec=103982.95 u/sec=22.75 ppl=7.497649 [2023-12-17:12:28:23:INFO:sockeye.training:__call__] E=23 B=93150 s/sec=2121.46 tok/sec=98094.94 u/sec=23.34 ppl=7.511366 [2023-12-17:12:28:25:INFO:sockeye.training:__call__] E=23 B=93200 s/sec=2038.07 tok/sec=99883.75 u/sec=22.95 ppl=7.522043 [2023-12-17:12:28:27:INFO:sockeye.training:__call__] E=23 B=93250 s/sec=2047.22 tok/sec=97150.66 u/sec=23.56 ppl=7.528836 [2023-12-17:12:28:29:INFO:sockeye.training:__call__] E=23 B=93300 s/sec=1907.19 tok/sec=100034.97 u/sec=23.60 ppl=7.538975 [2023-12-17:12:28:32:INFO:sockeye.training:__call__] E=23 B=93350 s/sec=2167.57 tok/sec=98540.94 u/sec=23.24 ppl=7.552663 [2023-12-17:12:28:34:INFO:sockeye.training:__call__] E=23 B=93400 s/sec=2213.41 tok/sec=95759.30 u/sec=23.81 ppl=7.563809 [2023-12-17:12:28:36:INFO:sockeye.training:__call__] E=23 B=93450 s/sec=2236.20 tok/sec=97257.63 u/sec=23.65 ppl=7.575928 [2023-12-17:12:28:38:INFO:sockeye.training:__call__] E=23 B=93500 s/sec=2116.35 tok/sec=99810.64 u/sec=23.49 ppl=7.585386 [2023-12-17:12:28:40:INFO:sockeye.training:__call__] E=23 B=93550 s/sec=1994.03 tok/sec=100833.73 u/sec=23.12 ppl=7.593392 [2023-12-17:12:28:42:INFO:sockeye.training:__call__] E=23 B=93600 s/sec=2077.26 tok/sec=99975.40 u/sec=23.18 ppl=7.603277 [2023-12-17:12:28:44:INFO:sockeye.training:__call__] E=23 B=93650 s/sec=2069.30 tok/sec=97268.65 u/sec=23.64 ppl=7.614107 [2023-12-17:12:28:46:INFO:sockeye.training:__call__] E=23 B=93700 s/sec=2063.36 tok/sec=100606.18 u/sec=23.07 ppl=7.624724 [2023-12-17:12:28:49:INFO:sockeye.training:__call__] E=23 B=93750 s/sec=2086.40 tok/sec=99158.29 u/sec=23.41 ppl=7.640766 [2023-12-17:12:28:51:INFO:sockeye.training:__call__] E=23 B=93800 s/sec=2121.88 tok/sec=99055.79 u/sec=23.23 ppl=7.653403 [2023-12-17:12:28:53:INFO:sockeye.training:__call__] E=23 B=93850 s/sec=2072.00 tok/sec=98541.45 u/sec=23.04 ppl=7.663720 [2023-12-17:12:28:55:INFO:sockeye.training:__call__] E=23 B=93900 s/sec=1753.24 tok/sec=102505.84 u/sec=22.83 ppl=7.665099 [2023-12-17:12:28:57:INFO:sockeye.training:__call__] E=23 B=93950 s/sec=1989.31 tok/sec=99083.21 u/sec=23.41 ppl=7.675669 [2023-12-17:12:28:59:INFO:sockeye.training:__call__] E=23 B=94000 s/sec=1784.38 tok/sec=102692.80 u/sec=22.76 ppl=7.675451 [2023-12-17:12:29:02:INFO:sockeye.training:__call__] E=23 B=94050 s/sec=2199.25 tok/sec=98845.97 u/sec=23.54 ppl=7.685232 [2023-12-17:12:29:04:INFO:sockeye.training:__call__] E=23 B=94100 s/sec=2083.57 tok/sec=100247.49 u/sec=23.05 ppl=7.697029 [2023-12-17:12:29:06:INFO:sockeye.training:__call__] E=23 B=94150 s/sec=2028.77 tok/sec=100345.29 u/sec=23.39 ppl=7.708191 [2023-12-17:12:29:08:INFO:sockeye.training:__call__] E=23 B=94200 s/sec=2077.64 tok/sec=100473.95 u/sec=23.35 ppl=7.718305 [2023-12-17:12:29:10:INFO:sockeye.training:__call__] E=23 B=94250 s/sec=1997.16 tok/sec=101681.89 u/sec=23.07 ppl=7.727353 [2023-12-17:12:29:12:INFO:sockeye.training:__call__] E=23 B=94300 s/sec=1903.98 tok/sec=101070.73 u/sec=23.20 ppl=7.734173 [2023-12-17:12:29:15:INFO:sockeye.training:__call__] E=23 B=94350 s/sec=1838.49 tok/sec=101475.61 u/sec=23.12 ppl=7.739384 [2023-12-17:12:29:17:INFO:sockeye.training:__call__] E=23 B=94400 s/sec=2239.14 tok/sec=96060.11 u/sec=23.60 ppl=7.747955 [2023-12-17:12:29:19:INFO:sockeye.training:__call__] E=23 B=94450 s/sec=2235.14 tok/sec=98228.41 u/sec=22.98 ppl=7.756737 [2023-12-17:12:29:21:INFO:sockeye.training:__call__] E=23 B=94500 s/sec=2205.25 tok/sec=98385.58 u/sec=23.52 ppl=7.765803 [2023-12-17:12:29:23:INFO:sockeye.training:__call__] E=23 B=94550 s/sec=1970.13 tok/sec=100159.06 u/sec=23.36 ppl=7.771147 [2023-12-17:12:29:25:INFO:sockeye.training:__call__] E=23 B=94600 s/sec=2055.17 tok/sec=101378.67 u/sec=23.19 ppl=7.781340 [2023-12-17:12:29:27:INFO:sockeye.training:__call__] E=23 B=94650 s/sec=2104.33 tok/sec=101474.79 u/sec=22.79 ppl=7.789559 [2023-12-17:12:29:29:INFO:sockeye.training:__call__] E=23 B=94700 s/sec=2359.10 tok/sec=94487.99 u/sec=24.21 ppl=7.800074 [2023-12-17:12:29:32:INFO:sockeye.training:__call__] E=23 B=94750 s/sec=2222.16 tok/sec=99158.44 u/sec=22.99 ppl=7.809241 [2023-12-17:12:29:34:INFO:sockeye.training:__call__] E=23 B=94800 s/sec=2310.82 tok/sec=94663.54 u/sec=23.87 ppl=7.819800 [2023-12-17:12:29:36:INFO:sockeye.training:__call__] E=23 B=94850 s/sec=2183.14 tok/sec=96317.37 u/sec=23.81 ppl=7.828236 [2023-12-17:12:29:38:INFO:sockeye.training:__call__] E=23 B=94900 s/sec=1961.99 tok/sec=101426.42 u/sec=22.75 ppl=7.833793 [2023-12-17:12:29:40:INFO:sockeye.training:__call__] E=23 B=94950 s/sec=2258.82 tok/sec=98281.10 u/sec=23.37 ppl=7.841647 [2023-12-17:12:29:42:INFO:sockeye.training:__call__] E=23 B=95000 s/sec=1933.49 tok/sec=101303.03 u/sec=22.89 ppl=7.849012 [2023-12-17:12:29:44:INFO:sockeye.training:__call__] E=23 B=95050 s/sec=2085.80 tok/sec=96457.93 u/sec=23.83 ppl=7.857116 [2023-12-17:12:29:47:INFO:sockeye.training:__call__] E=23 B=95100 s/sec=1996.19 tok/sec=102140.87 u/sec=22.77 ppl=7.861762 [2023-12-17:12:29:49:INFO:sockeye.training:__call__] E=23 B=95150 s/sec=2030.93 tok/sec=100745.77 u/sec=22.75 ppl=7.869623 [2023-12-17:12:29:51:INFO:sockeye.training:__call__] E=23 B=95200 s/sec=2065.01 tok/sec=98467.94 u/sec=23.17 ppl=7.876741 [2023-12-17:12:29:53:INFO:sockeye.training:__call__] E=23 B=95250 s/sec=2252.06 tok/sec=95517.21 u/sec=23.46 ppl=7.886567 [2023-12-17:12:29:55:INFO:sockeye.training:__call__] E=23 B=95300 s/sec=1995.69 tok/sec=101358.86 u/sec=22.89 ppl=7.893557 [2023-12-17:12:29:58:INFO:sockeye.training:__call__] E=23 B=95350 s/sec=2002.22 tok/sec=105705.89 u/sec=22.19 ppl=7.897726 [2023-12-17:12:30:00:INFO:sockeye.training:__call__] E=23 B=95400 s/sec=2173.20 tok/sec=96214.35 u/sec=23.79 ppl=7.905857 [2023-12-17:12:30:02:INFO:sockeye.training:__call__] E=23 B=95450 s/sec=1925.11 tok/sec=99522.34 u/sec=22.79 ppl=7.913595 [2023-12-17:12:30:04:INFO:sockeye.training:__call__] E=23 B=95500 s/sec=2160.78 tok/sec=96440.44 u/sec=23.20 ppl=7.922028 [2023-12-17:12:30:06:INFO:sockeye.training:__call__] E=23 B=95550 s/sec=2107.28 tok/sec=98514.54 u/sec=23.39 ppl=7.928722 [2023-12-17:12:30:08:INFO:sockeye.training:__call__] E=23 B=95600 s/sec=1974.03 tok/sec=99948.00 u/sec=23.32 ppl=7.937309 [2023-12-17:12:30:11:INFO:sockeye.training:__call__] E=23 B=95650 s/sec=1882.49 tok/sec=102904.53 u/sec=22.85 ppl=7.944393 [2023-12-17:12:30:13:INFO:sockeye.training:__call__] E=23 B=95700 s/sec=2141.42 tok/sec=98051.04 u/sec=23.28 ppl=7.953718 [2023-12-17:12:30:15:INFO:sockeye.training:__call__] E=23 B=95750 s/sec=2010.67 tok/sec=100162.61 u/sec=23.19 ppl=7.960701 [2023-12-17:12:30:17:INFO:sockeye.training:__call__] E=23 B=95800 s/sec=1967.86 tok/sec=101354.03 u/sec=22.73 ppl=7.967118 [2023-12-17:12:30:19:INFO:sockeye.training:__call__] E=23 B=95850 s/sec=2378.55 tok/sec=96111.64 u/sec=23.67 ppl=7.976602 [2023-12-17:12:30:21:INFO:sockeye.training:__call__] E=24 B=95900 s/sec=1882.10 tok/sec=96324.66 u/sec=21.54 ppl=7.976820 [2023-12-17:12:30:24:INFO:sockeye.training:__call__] E=24 B=95950 s/sec=2168.27 tok/sec=95373.74 u/sec=23.45 ppl=7.964934 [2023-12-17:12:30:26:INFO:sockeye.training:__call__] E=24 B=96000 s/sec=2131.10 tok/sec=99708.81 u/sec=23.20 ppl=7.952494 [2023-12-17:12:30:26:INFO:sockeye.training:_create_checkpoint] Checkpoint [24] Updates=96000 Epoch=24 Samples=8555896 Time-cost=172.428 Updates/sec=23.198 [2023-12-17:12:30:26:INFO:sockeye.training:_create_checkpoint] Checkpoint [24] Train-ppl=7.952494 [2023-12-17:12:30:28:INFO:sockeye.training:_evaluate] Checkpoint [24] Validation-ppl=29.021747 Validation-bleu=0.188692 Validation-chrf=32.780032 Validation-rouge1=0.172396 Validation-rouge2=0.087062 Validation-rougel=0.166430 Validation-length-ratio-mse=1.267278 Validation-ter=93.355345 Validation-avg-sec-per-sent=0.003228 Validation-decode-walltime=1.613806 [2023-12-17:12:30:28:INFO:sockeye.training:_determine_improvement] Validation-chrf improved to 32.780032 (delta=0.796795). [2023-12-17:12:30:28:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 1.861041 > 0.000000 over 20 checkpoints [2023-12-17:12:30:28:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [24] Learning-rate=0.000162 [2023-12-17:12:30:29:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:30:29:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:30:29:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00024" [2023-12-17:12:30:29:INFO:sockeye.training:_update_best_params] '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best' now points to 'params.00024' [2023-12-17:12:30:30:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl" [2023-12-17:12:30:30:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/lr_scheduler_best.pkl' [2023-12-17:12:30:32:INFO:sockeye.training:__call__] E=24 B=96050 s/sec=685.01 tok/sec=36158.10 u/sec=8.08 ppl=7.045107 [2023-12-17:12:30:34:INFO:sockeye.training:__call__] E=24 B=96100 s/sec=2164.97 tok/sec=98859.70 u/sec=23.25 ppl=7.097912 [2023-12-17:12:30:36:INFO:sockeye.training:__call__] E=24 B=96150 s/sec=1974.35 tok/sec=97451.28 u/sec=23.55 ppl=7.050912 [2023-12-17:12:30:38:INFO:sockeye.training:__call__] E=24 B=96200 s/sec=2116.39 tok/sec=100521.09 u/sec=23.12 ppl=7.070229 [2023-12-17:12:30:41:INFO:sockeye.training:__call__] E=24 B=96250 s/sec=2214.18 tok/sec=95821.77 u/sec=23.62 ppl=7.088746 [2023-12-17:12:30:43:INFO:sockeye.training:__call__] E=24 B=96300 s/sec=1867.32 tok/sec=102887.28 u/sec=22.66 ppl=7.069587 [2023-12-17:12:30:45:INFO:sockeye.training:__call__] E=24 B=96350 s/sec=2116.12 tok/sec=99277.34 u/sec=23.00 ppl=7.078230 [2023-12-17:12:30:47:INFO:sockeye.training:__call__] E=24 B=96400 s/sec=2113.74 tok/sec=101253.89 u/sec=22.94 ppl=7.094035 [2023-12-17:12:30:49:INFO:sockeye.training:__call__] E=24 B=96450 s/sec=1703.95 tok/sec=103013.37 u/sec=22.47 ppl=7.081947 [2023-12-17:12:30:51:INFO:sockeye.training:__call__] E=24 B=96500 s/sec=2009.13 tok/sec=101737.24 u/sec=22.87 ppl=7.106112 [2023-12-17:12:30:54:INFO:sockeye.training:__call__] E=24 B=96550 s/sec=2136.47 tok/sec=100886.16 u/sec=23.22 ppl=7.116306 [2023-12-17:12:30:56:INFO:sockeye.training:__call__] E=24 B=96600 s/sec=2054.94 tok/sec=100251.24 u/sec=23.35 ppl=7.127340 [2023-12-17:12:30:58:INFO:sockeye.training:__call__] E=24 B=96650 s/sec=1990.06 tok/sec=100050.63 u/sec=22.82 ppl=7.138362 [2023-12-17:12:31:00:INFO:sockeye.training:__call__] E=24 B=96700 s/sec=2140.52 tok/sec=100881.62 u/sec=23.11 ppl=7.156782 [2023-12-17:12:31:02:INFO:sockeye.training:__call__] E=24 B=96750 s/sec=2199.06 tok/sec=96983.56 u/sec=23.41 ppl=7.162682 [2023-12-17:12:31:04:INFO:sockeye.training:__call__] E=24 B=96800 s/sec=2180.87 tok/sec=97860.92 u/sec=22.87 ppl=7.182753 [2023-12-17:12:31:07:INFO:sockeye.training:__call__] E=24 B=96850 s/sec=2047.26 tok/sec=100944.40 u/sec=22.97 ppl=7.188651 [2023-12-17:12:31:09:INFO:sockeye.training:__call__] E=24 B=96900 s/sec=1673.97 tok/sec=104369.10 u/sec=22.50 ppl=7.187093 [2023-12-17:12:31:11:INFO:sockeye.training:__call__] E=24 B=96950 s/sec=2057.23 tok/sec=100398.87 u/sec=23.04 ppl=7.189487 [2023-12-17:12:31:13:INFO:sockeye.training:__call__] E=24 B=97000 s/sec=1995.86 tok/sec=101438.18 u/sec=23.06 ppl=7.198443 [2023-12-17:12:31:15:INFO:sockeye.training:__call__] E=24 B=97050 s/sec=2301.57 tok/sec=96181.33 u/sec=23.66 ppl=7.212840 [2023-12-17:12:31:17:INFO:sockeye.training:__call__] E=24 B=97100 s/sec=1688.85 tok/sec=103197.87 u/sec=23.15 ppl=7.214406 [2023-12-17:12:31:20:INFO:sockeye.training:__call__] E=24 B=97150 s/sec=2158.33 tok/sec=96901.48 u/sec=23.26 ppl=7.230911 [2023-12-17:12:31:22:INFO:sockeye.training:__call__] E=24 B=97200 s/sec=1989.40 tok/sec=101738.49 u/sec=23.24 ppl=7.241206 [2023-12-17:12:31:24:INFO:sockeye.training:__call__] E=24 B=97250 s/sec=2108.81 tok/sec=96674.08 u/sec=23.54 ppl=7.252981 [2023-12-17:12:31:26:INFO:sockeye.training:__call__] E=24 B=97300 s/sec=2086.09 tok/sec=99714.24 u/sec=23.32 ppl=7.261254 [2023-12-17:12:31:28:INFO:sockeye.training:__call__] E=24 B=97350 s/sec=2062.08 tok/sec=98009.22 u/sec=23.69 ppl=7.267883 [2023-12-17:12:31:30:INFO:sockeye.training:__call__] E=24 B=97400 s/sec=2194.34 tok/sec=98511.43 u/sec=22.86 ppl=7.281523 [2023-12-17:12:31:33:INFO:sockeye.training:__call__] E=24 B=97450 s/sec=2091.14 tok/sec=100666.30 u/sec=22.77 ppl=7.290224 [2023-12-17:12:31:35:INFO:sockeye.training:__call__] E=24 B=97500 s/sec=2274.23 tok/sec=97901.65 u/sec=23.65 ppl=7.304345 [2023-12-17:12:31:37:INFO:sockeye.training:__call__] E=24 B=97550 s/sec=2032.04 tok/sec=103256.48 u/sec=22.68 ppl=7.315236 [2023-12-17:12:31:39:INFO:sockeye.training:__call__] E=24 B=97600 s/sec=2055.15 tok/sec=102046.96 u/sec=22.90 ppl=7.324075 [2023-12-17:12:31:41:INFO:sockeye.training:__call__] E=24 B=97650 s/sec=2179.11 tok/sec=97558.16 u/sec=23.40 ppl=7.334961 [2023-12-17:12:31:43:INFO:sockeye.training:__call__] E=24 B=97700 s/sec=2002.16 tok/sec=100810.93 u/sec=23.13 ppl=7.341292 [2023-12-17:12:31:45:INFO:sockeye.training:__call__] E=24 B=97750 s/sec=2213.30 tok/sec=97369.98 u/sec=23.65 ppl=7.351472 [2023-12-17:12:31:48:INFO:sockeye.training:__call__] E=24 B=97800 s/sec=2164.16 tok/sec=99135.71 u/sec=23.61 ppl=7.362552 [2023-12-17:12:31:50:INFO:sockeye.training:__call__] E=24 B=97850 s/sec=2163.42 tok/sec=96982.32 u/sec=23.93 ppl=7.372963 [2023-12-17:12:31:52:INFO:sockeye.training:__call__] E=24 B=97900 s/sec=1986.61 tok/sec=99359.74 u/sec=22.82 ppl=7.380966 [2023-12-17:12:31:54:INFO:sockeye.training:__call__] E=24 B=97950 s/sec=1934.90 tok/sec=98849.14 u/sec=22.99 ppl=7.385508 [2023-12-17:12:31:56:INFO:sockeye.training:__call__] E=24 B=98000 s/sec=2031.33 tok/sec=100314.79 u/sec=22.88 ppl=7.394004 [2023-12-17:12:31:58:INFO:sockeye.training:__call__] E=24 B=98050 s/sec=2106.34 tok/sec=96606.82 u/sec=23.42 ppl=7.404017 [2023-12-17:12:32:01:INFO:sockeye.training:__call__] E=24 B=98100 s/sec=1811.05 tok/sec=101154.10 u/sec=22.96 ppl=7.407704 [2023-12-17:12:32:03:INFO:sockeye.training:__call__] E=24 B=98150 s/sec=1910.84 tok/sec=102330.42 u/sec=22.84 ppl=7.414746 [2023-12-17:12:32:05:INFO:sockeye.training:__call__] E=24 B=98200 s/sec=2284.30 tok/sec=95718.78 u/sec=23.91 ppl=7.426424 [2023-12-17:12:32:07:INFO:sockeye.training:__call__] E=24 B=98250 s/sec=1992.23 tok/sec=107220.67 u/sec=22.04 ppl=7.433461 [2023-12-17:12:32:09:INFO:sockeye.training:__call__] E=24 B=98300 s/sec=2255.08 tok/sec=96611.68 u/sec=23.73 ppl=7.443245 [2023-12-17:12:32:11:INFO:sockeye.training:__call__] E=24 B=98350 s/sec=2031.53 tok/sec=98156.38 u/sec=23.91 ppl=7.451296 [2023-12-17:12:32:13:INFO:sockeye.training:__call__] E=24 B=98400 s/sec=2088.86 tok/sec=100028.85 u/sec=23.11 ppl=7.463285 [2023-12-17:12:32:16:INFO:sockeye.training:__call__] E=24 B=98450 s/sec=2259.20 tok/sec=97918.94 u/sec=23.69 ppl=7.472140 [2023-12-17:12:32:18:INFO:sockeye.training:__call__] E=24 B=98500 s/sec=2056.20 tok/sec=96407.91 u/sec=23.54 ppl=7.480609 [2023-12-17:12:32:20:INFO:sockeye.training:__call__] E=24 B=98550 s/sec=1895.12 tok/sec=101624.94 u/sec=22.69 ppl=7.485402 [2023-12-17:12:32:22:INFO:sockeye.training:__call__] E=24 B=98600 s/sec=2096.67 tok/sec=100341.94 u/sec=23.32 ppl=7.492966 [2023-12-17:12:32:24:INFO:sockeye.training:__call__] E=24 B=98650 s/sec=2000.07 tok/sec=104625.84 u/sec=22.60 ppl=7.498083 [2023-12-17:12:32:26:INFO:sockeye.training:__call__] E=24 B=98700 s/sec=2091.64 tok/sec=98145.42 u/sec=23.47 ppl=7.505832 [2023-12-17:12:32:28:INFO:sockeye.training:__call__] E=24 B=98750 s/sec=2204.79 tok/sec=97703.22 u/sec=23.52 ppl=7.516422 [2023-12-17:12:32:31:INFO:sockeye.training:__call__] E=24 B=98800 s/sec=2231.78 tok/sec=96296.50 u/sec=23.97 ppl=7.525286 [2023-12-17:12:32:33:INFO:sockeye.training:__call__] E=24 B=98850 s/sec=2113.14 tok/sec=95350.99 u/sec=24.23 ppl=7.532347 [2023-12-17:12:32:35:INFO:sockeye.training:__call__] E=24 B=98900 s/sec=2168.01 tok/sec=98513.24 u/sec=23.44 ppl=7.540274 [2023-12-17:12:32:37:INFO:sockeye.training:__call__] E=24 B=98950 s/sec=2061.73 tok/sec=96325.43 u/sec=23.69 ppl=7.545046 [2023-12-17:12:32:39:INFO:sockeye.training:__call__] E=24 B=99000 s/sec=1999.83 tok/sec=101412.22 u/sec=22.77 ppl=7.550136 [2023-12-17:12:32:41:INFO:sockeye.training:__call__] E=24 B=99050 s/sec=2245.91 tok/sec=95825.45 u/sec=23.75 ppl=7.559895 [2023-12-17:12:32:43:INFO:sockeye.training:__call__] E=24 B=99100 s/sec=2038.00 tok/sec=97400.36 u/sec=23.63 ppl=7.564138 [2023-12-17:12:32:45:INFO:sockeye.training:__call__] E=24 B=99150 s/sec=2101.72 tok/sec=99877.96 u/sec=23.09 ppl=7.570593 [2023-12-17:12:32:48:INFO:sockeye.training:__call__] E=24 B=99200 s/sec=1904.85 tok/sec=101324.41 u/sec=23.21 ppl=7.574962 [2023-12-17:12:32:50:INFO:sockeye.training:__call__] E=24 B=99250 s/sec=2136.91 tok/sec=97584.09 u/sec=23.47 ppl=7.582547 [2023-12-17:12:32:52:INFO:sockeye.training:__call__] E=24 B=99300 s/sec=2044.57 tok/sec=99062.84 u/sec=23.28 ppl=7.589976 [2023-12-17:12:32:54:INFO:sockeye.training:__call__] E=24 B=99350 s/sec=1926.51 tok/sec=101465.35 u/sec=23.11 ppl=7.597073 [2023-12-17:12:32:56:INFO:sockeye.training:__call__] E=24 B=99400 s/sec=1861.06 tok/sec=104598.15 u/sec=22.76 ppl=7.602206 [2023-12-17:12:32:58:INFO:sockeye.training:__call__] E=24 B=99450 s/sec=2118.37 tok/sec=99726.20 u/sec=23.15 ppl=7.608583 [2023-12-17:12:33:01:INFO:sockeye.training:__call__] E=24 B=99500 s/sec=2379.80 tok/sec=95404.67 u/sec=23.72 ppl=7.617583 [2023-12-17:12:33:03:INFO:sockeye.training:__call__] E=24 B=99550 s/sec=2056.55 tok/sec=99847.14 u/sec=23.29 ppl=7.623657 [2023-12-17:12:33:05:INFO:sockeye.training:__call__] E=24 B=99600 s/sec=2021.59 tok/sec=96945.77 u/sec=23.53 ppl=7.629213 [2023-12-17:12:33:07:INFO:sockeye.training:__call__] E=24 B=99650 s/sec=1914.47 tok/sec=100070.96 u/sec=23.14 ppl=7.635230 [2023-12-17:12:33:09:INFO:sockeye.training:__call__] E=24 B=99700 s/sec=2291.38 tok/sec=97694.93 u/sec=23.59 ppl=7.641862 [2023-12-17:12:33:11:INFO:sockeye.training:__call__] E=24 B=99750 s/sec=2186.01 tok/sec=98264.96 u/sec=23.56 ppl=7.650465 [2023-12-17:12:33:13:INFO:sockeye.training:__call__] E=24 B=99800 s/sec=2142.76 tok/sec=99245.14 u/sec=23.29 ppl=7.657914 [2023-12-17:12:33:16:INFO:sockeye.training:__call__] E=24 B=99850 s/sec=2096.54 tok/sec=97315.40 u/sec=23.36 ppl=7.663302 [2023-12-17:12:33:18:INFO:sockeye.training:__call__] E=25 B=99900 s/sec=1936.49 tok/sec=93439.03 u/sec=22.41 ppl=7.660682 [2023-12-17:12:33:20:INFO:sockeye.training:__call__] E=25 B=99950 s/sec=1956.62 tok/sec=99435.21 u/sec=23.43 ppl=7.646830 [2023-12-17:12:33:22:INFO:sockeye.training:__call__] E=25 B=100000 s/sec=1920.78 tok/sec=102899.23 u/sec=22.36 ppl=7.635172 [2023-12-17:12:33:22:INFO:sockeye.training:_create_checkpoint] Checkpoint [25] Updates=100000 Epoch=25 Samples=8911768 Time-cost=172.360 Updates/sec=23.207 [2023-12-17:12:33:22:INFO:sockeye.training:_create_checkpoint] Checkpoint [25] Train-ppl=7.635172 [2023-12-17:12:33:25:INFO:sockeye.training:_evaluate] Checkpoint [25] Validation-ppl=29.364699 Validation-bleu=0.157933 Validation-chrf=30.289782 Validation-rouge1=0.154094 Validation-rouge2=0.070716 Validation-rougel=0.146999 Validation-length-ratio-mse=1.275279 Validation-ter=96.244325 Validation-avg-sec-per-sent=0.003349 Validation-decode-walltime=1.674365 [2023-12-17:12:33:25:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 1 checkpoints, best so far: 32.780032 [2023-12-17:12:33:25:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:12:33:25:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [25] Learning-rate=0.000162 [2023-12-17:12:33:25:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:33:25:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:33:26:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00025" [2023-12-17:12:33:28:INFO:sockeye.training:__call__] E=25 B=100050 s/sec=776.43 tok/sec=36827.15 u/sec=8.62 ppl=6.780704 [2023-12-17:12:33:30:INFO:sockeye.training:__call__] E=25 B=100100 s/sec=1937.84 tok/sec=98691.50 u/sec=23.34 ppl=6.691622 [2023-12-17:12:33:32:INFO:sockeye.training:__call__] E=25 B=100150 s/sec=1993.64 tok/sec=97813.01 u/sec=23.64 ppl=6.732342 [2023-12-17:12:33:34:INFO:sockeye.training:__call__] E=25 B=100200 s/sec=2118.25 tok/sec=97939.92 u/sec=23.35 ppl=6.759120 [2023-12-17:12:33:36:INFO:sockeye.training:__call__] E=25 B=100250 s/sec=2361.16 tok/sec=96144.30 u/sec=23.69 ppl=6.790843 [2023-12-17:12:33:39:INFO:sockeye.training:__call__] E=25 B=100300 s/sec=2125.60 tok/sec=98533.89 u/sec=22.94 ppl=6.817435 [2023-12-17:12:33:41:INFO:sockeye.training:__call__] E=25 B=100350 s/sec=1923.53 tok/sec=100683.08 u/sec=22.90 ppl=6.809359 [2023-12-17:12:33:43:INFO:sockeye.training:__call__] E=25 B=100400 s/sec=2094.60 tok/sec=99194.03 u/sec=23.38 ppl=6.825815 [2023-12-17:12:33:45:INFO:sockeye.training:__call__] E=25 B=100450 s/sec=2078.46 tok/sec=99912.39 u/sec=22.87 ppl=6.831058 [2023-12-17:12:33:47:INFO:sockeye.training:__call__] E=25 B=100500 s/sec=2407.44 tok/sec=94889.01 u/sec=23.92 ppl=6.873054 [2023-12-17:12:33:49:INFO:sockeye.training:__call__] E=25 B=100550 s/sec=2280.53 tok/sec=97586.91 u/sec=23.80 ppl=6.889689 [2023-12-17:12:33:51:INFO:sockeye.training:__call__] E=25 B=100600 s/sec=1915.39 tok/sec=102564.96 u/sec=22.93 ppl=6.901435 [2023-12-17:12:33:54:INFO:sockeye.training:__call__] E=25 B=100650 s/sec=2096.84 tok/sec=99701.79 u/sec=23.11 ppl=6.911178 [2023-12-17:12:33:56:INFO:sockeye.training:__call__] E=25 B=100700 s/sec=2173.44 tok/sec=99042.29 u/sec=22.68 ppl=6.926534 [2023-12-17:12:33:58:INFO:sockeye.training:__call__] E=25 B=100750 s/sec=2070.73 tok/sec=98709.68 u/sec=23.28 ppl=6.938742 [2023-12-17:12:34:00:INFO:sockeye.training:__call__] E=25 B=100800 s/sec=1901.62 tok/sec=102512.19 u/sec=22.86 ppl=6.945803 [2023-12-17:12:34:02:INFO:sockeye.training:__call__] E=25 B=100850 s/sec=1812.99 tok/sec=103607.79 u/sec=22.57 ppl=6.947354 [2023-12-17:12:34:05:INFO:sockeye.training:__call__] E=25 B=100900 s/sec=1967.45 tok/sec=99733.89 u/sec=23.24 ppl=6.956008 [2023-12-17:12:34:07:INFO:sockeye.training:__call__] E=25 B=100950 s/sec=2071.43 tok/sec=97248.92 u/sec=23.33 ppl=6.967165 [2023-12-17:12:34:09:INFO:sockeye.training:__call__] E=25 B=101000 s/sec=2125.40 tok/sec=97670.45 u/sec=23.59 ppl=6.977736 [2023-12-17:12:34:11:INFO:sockeye.training:__call__] E=25 B=101050 s/sec=2196.24 tok/sec=100734.20 u/sec=23.46 ppl=6.982303 [2023-12-17:12:34:13:INFO:sockeye.training:__call__] E=25 B=101100 s/sec=2170.01 tok/sec=99170.70 u/sec=23.34 ppl=6.996416 [2023-12-17:12:34:15:INFO:sockeye.training:__call__] E=25 B=101150 s/sec=2006.88 tok/sec=100134.55 u/sec=23.40 ppl=7.004556 [2023-12-17:12:34:17:INFO:sockeye.training:__call__] E=25 B=101200 s/sec=2166.19 tok/sec=95632.36 u/sec=23.79 ppl=7.021227 [2023-12-17:12:34:19:INFO:sockeye.training:__call__] E=25 B=101250 s/sec=2056.37 tok/sec=101034.61 u/sec=23.33 ppl=7.031456 [2023-12-17:12:34:22:INFO:sockeye.training:__call__] E=25 B=101300 s/sec=1927.07 tok/sec=102707.76 u/sec=23.12 ppl=7.031677 [2023-12-17:12:34:24:INFO:sockeye.training:__call__] E=25 B=101350 s/sec=1992.83 tok/sec=101232.60 u/sec=22.81 ppl=7.035524 [2023-12-17:12:34:26:INFO:sockeye.training:__call__] E=25 B=101400 s/sec=1857.34 tok/sec=109046.53 u/sec=21.50 ppl=7.039307 [2023-12-17:12:34:28:INFO:sockeye.training:__call__] E=25 B=101450 s/sec=2012.50 tok/sec=101283.83 u/sec=22.87 ppl=7.049417 [2023-12-17:12:34:30:INFO:sockeye.training:__call__] E=25 B=101500 s/sec=2177.88 tok/sec=96662.14 u/sec=23.31 ppl=7.061490 [2023-12-17:12:34:33:INFO:sockeye.training:__call__] E=25 B=101550 s/sec=2268.68 tok/sec=98097.19 u/sec=23.44 ppl=7.075727 [2023-12-17:12:34:35:INFO:sockeye.training:__call__] E=25 B=101600 s/sec=1975.43 tok/sec=102060.38 u/sec=22.99 ppl=7.083187 [2023-12-17:12:34:37:INFO:sockeye.training:__call__] E=25 B=101650 s/sec=2268.67 tok/sec=96877.72 u/sec=23.83 ppl=7.095295 [2023-12-17:12:34:39:INFO:sockeye.training:__call__] E=25 B=101700 s/sec=2080.85 tok/sec=100424.82 u/sec=23.60 ppl=7.102428 [2023-12-17:12:34:41:INFO:sockeye.training:__call__] E=25 B=101750 s/sec=2142.99 tok/sec=99637.66 u/sec=23.33 ppl=7.112371 [2023-12-17:12:34:43:INFO:sockeye.training:__call__] E=25 B=101800 s/sec=2228.08 tok/sec=97173.07 u/sec=23.64 ppl=7.121199 [2023-12-17:12:34:45:INFO:sockeye.training:__call__] E=25 B=101850 s/sec=2170.14 tok/sec=96573.28 u/sec=23.26 ppl=7.131660 [2023-12-17:12:34:48:INFO:sockeye.training:__call__] E=25 B=101900 s/sec=2034.04 tok/sec=100882.68 u/sec=23.07 ppl=7.139734 [2023-12-17:12:34:50:INFO:sockeye.training:__call__] E=25 B=101950 s/sec=2105.88 tok/sec=97906.43 u/sec=23.63 ppl=7.145896 [2023-12-17:12:34:52:INFO:sockeye.training:__call__] E=25 B=102000 s/sec=2007.70 tok/sec=95887.29 u/sec=23.54 ppl=7.154744 [2023-12-17:12:34:54:INFO:sockeye.training:__call__] E=25 B=102050 s/sec=2148.75 tok/sec=99316.48 u/sec=23.15 ppl=7.166014 [2023-12-17:12:34:56:INFO:sockeye.training:__call__] E=25 B=102100 s/sec=2026.84 tok/sec=99846.83 u/sec=23.24 ppl=7.171026 [2023-12-17:12:34:58:INFO:sockeye.training:__call__] E=25 B=102150 s/sec=2023.34 tok/sec=97911.46 u/sec=23.77 ppl=7.178161 [2023-12-17:12:35:00:INFO:sockeye.training:__call__] E=25 B=102200 s/sec=2107.29 tok/sec=98818.66 u/sec=23.35 ppl=7.183614 [2023-12-17:12:35:02:INFO:sockeye.training:__call__] E=25 B=102250 s/sec=2104.44 tok/sec=96621.17 u/sec=23.83 ppl=7.191474 [2023-12-17:12:35:05:INFO:sockeye.training:__call__] E=25 B=102300 s/sec=1791.75 tok/sec=104718.06 u/sec=22.22 ppl=7.195700 [2023-12-17:12:35:07:INFO:sockeye.training:__call__] E=25 B=102350 s/sec=2137.95 tok/sec=99682.77 u/sec=22.96 ppl=7.205597 [2023-12-17:12:35:09:INFO:sockeye.training:__call__] E=25 B=102400 s/sec=2165.95 tok/sec=97291.77 u/sec=23.92 ppl=7.211949 [2023-12-17:12:35:11:INFO:sockeye.training:__call__] E=25 B=102450 s/sec=2101.90 tok/sec=99437.88 u/sec=23.17 ppl=7.220029 [2023-12-17:12:35:13:INFO:sockeye.training:__call__] E=25 B=102500 s/sec=2083.74 tok/sec=97902.04 u/sec=23.30 ppl=7.227154 [2023-12-17:12:35:15:INFO:sockeye.training:__call__] E=25 B=102550 s/sec=2161.03 tok/sec=95633.07 u/sec=23.78 ppl=7.234617 [2023-12-17:12:35:17:INFO:sockeye.training:__call__] E=25 B=102600 s/sec=2329.53 tok/sec=93517.70 u/sec=23.91 ppl=7.241856 [2023-12-17:12:35:20:INFO:sockeye.training:__call__] E=25 B=102650 s/sec=1751.11 tok/sec=104607.30 u/sec=22.43 ppl=7.243086 [2023-12-17:12:35:22:INFO:sockeye.training:__call__] E=25 B=102700 s/sec=2138.55 tok/sec=99069.07 u/sec=23.12 ppl=7.252809 [2023-12-17:12:35:24:INFO:sockeye.training:__call__] E=25 B=102750 s/sec=2058.14 tok/sec=101632.66 u/sec=22.89 ppl=7.257495 [2023-12-17:12:35:26:INFO:sockeye.training:__call__] E=25 B=102800 s/sec=2431.19 tok/sec=94954.12 u/sec=23.89 ppl=7.267142 [2023-12-17:12:35:28:INFO:sockeye.training:__call__] E=25 B=102850 s/sec=2084.97 tok/sec=98818.68 u/sec=23.44 ppl=7.272386 [2023-12-17:12:35:30:INFO:sockeye.training:__call__] E=25 B=102900 s/sec=2097.90 tok/sec=97276.72 u/sec=23.46 ppl=7.279359 [2023-12-17:12:35:33:INFO:sockeye.training:__call__] E=25 B=102950 s/sec=1805.07 tok/sec=100814.66 u/sec=23.36 ppl=7.284813 [2023-12-17:12:35:35:INFO:sockeye.training:__call__] E=25 B=103000 s/sec=1785.40 tok/sec=101165.49 u/sec=23.10 ppl=7.289617 [2023-12-17:12:35:37:INFO:sockeye.training:__call__] E=25 B=103050 s/sec=2121.03 tok/sec=97064.27 u/sec=23.46 ppl=7.295165 [2023-12-17:12:35:39:INFO:sockeye.training:__call__] E=25 B=103100 s/sec=2085.21 tok/sec=99089.78 u/sec=22.99 ppl=7.302231 [2023-12-17:12:35:41:INFO:sockeye.training:__call__] E=25 B=103150 s/sec=2114.53 tok/sec=100408.81 u/sec=22.98 ppl=7.309158 [2023-12-17:12:35:43:INFO:sockeye.training:__call__] E=25 B=103200 s/sec=1899.63 tok/sec=104014.40 u/sec=22.53 ppl=7.315455 [2023-12-17:12:35:46:INFO:sockeye.training:__call__] E=25 B=103250 s/sec=1949.05 tok/sec=102556.79 u/sec=22.98 ppl=7.322729 [2023-12-17:12:35:48:INFO:sockeye.training:__call__] E=25 B=103300 s/sec=1862.09 tok/sec=100995.97 u/sec=23.09 ppl=7.329313 [2023-12-17:12:35:50:INFO:sockeye.training:__call__] E=25 B=103350 s/sec=2143.63 tok/sec=98818.25 u/sec=23.59 ppl=7.337860 [2023-12-17:12:35:52:INFO:sockeye.training:__call__] E=25 B=103400 s/sec=2310.26 tok/sec=98182.34 u/sec=23.36 ppl=7.345265 [2023-12-17:12:35:54:INFO:sockeye.training:__call__] E=25 B=103450 s/sec=2055.33 tok/sec=97444.73 u/sec=23.66 ppl=7.352246 [2023-12-17:12:35:56:INFO:sockeye.training:__call__] E=25 B=103500 s/sec=2015.63 tok/sec=102115.70 u/sec=22.90 ppl=7.357813 [2023-12-17:12:35:58:INFO:sockeye.training:__call__] E=25 B=103550 s/sec=2198.68 tok/sec=99763.92 u/sec=23.49 ppl=7.365727 [2023-12-17:12:36:01:INFO:sockeye.training:__call__] E=25 B=103600 s/sec=2050.42 tok/sec=100385.15 u/sec=23.26 ppl=7.373078 [2023-12-17:12:36:03:INFO:sockeye.training:__call__] E=25 B=103650 s/sec=1992.18 tok/sec=100809.98 u/sec=22.89 ppl=7.378618 [2023-12-17:12:36:05:INFO:sockeye.training:__call__] E=25 B=103700 s/sec=2059.41 tok/sec=96691.02 u/sec=23.45 ppl=7.382790 [2023-12-17:12:36:07:INFO:sockeye.training:__call__] E=25 B=103750 s/sec=2259.47 tok/sec=95629.64 u/sec=23.58 ppl=7.391073 [2023-12-17:12:36:09:INFO:sockeye.training:__call__] E=25 B=103800 s/sec=1991.74 tok/sec=100973.83 u/sec=23.05 ppl=7.396210 [2023-12-17:12:36:11:INFO:sockeye.training:__call__] E=25 B=103850 s/sec=1956.16 tok/sec=101059.50 u/sec=22.94 ppl=7.402247 [2023-12-17:12:36:14:INFO:sockeye.training:__call__] E=26 B=103900 s/sec=1899.20 tok/sec=95478.17 u/sec=22.19 ppl=7.398830 [2023-12-17:12:36:16:INFO:sockeye.training:__call__] E=26 B=103950 s/sec=1965.49 tok/sec=101167.23 u/sec=23.18 ppl=7.385925 [2023-12-17:12:36:18:INFO:sockeye.training:__call__] E=26 B=104000 s/sec=2142.07 tok/sec=100214.96 u/sec=23.12 ppl=7.373765 [2023-12-17:12:36:18:INFO:sockeye.training:_create_checkpoint] Checkpoint [26] Updates=104000 Epoch=26 Samples=9268368 Time-cost=172.240 Updates/sec=23.223 [2023-12-17:12:36:18:INFO:sockeye.training:_create_checkpoint] Checkpoint [26] Train-ppl=7.373765 [2023-12-17:12:36:21:INFO:sockeye.training:_evaluate] Checkpoint [26] Validation-ppl=29.366776 Validation-bleu=0.169713 Validation-chrf=32.052151 Validation-rouge1=0.169844 Validation-rouge2=0.086825 Validation-rougel=0.163671 Validation-length-ratio-mse=1.338613 Validation-ter=97.936442 Validation-avg-sec-per-sent=0.003355 Validation-decode-walltime=1.677655 [2023-12-17:12:36:21:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 2 checkpoints, best so far: 32.780032 [2023-12-17:12:36:21:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:12:36:21:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [26] Learning-rate=0.000162 [2023-12-17:12:36:21:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:36:21:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:36:22:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00026" [2023-12-17:12:36:24:INFO:sockeye.training:__call__] E=26 B=104050 s/sec=698.13 tok/sec=38182.81 u/sec=8.49 ppl=6.509015 [2023-12-17:12:36:26:INFO:sockeye.training:__call__] E=26 B=104100 s/sec=1994.06 tok/sec=98660.30 u/sec=22.78 ppl=6.529709 [2023-12-17:12:36:28:INFO:sockeye.training:__call__] E=26 B=104150 s/sec=2146.83 tok/sec=97660.52 u/sec=23.25 ppl=6.578500 [2023-12-17:12:36:30:INFO:sockeye.training:__call__] E=26 B=104200 s/sec=2142.34 tok/sec=99794.03 u/sec=23.05 ppl=6.577669 [2023-12-17:12:36:33:INFO:sockeye.training:__call__] E=26 B=104250 s/sec=2033.60 tok/sec=102017.93 u/sec=22.94 ppl=6.574667 [2023-12-17:12:36:35:INFO:sockeye.training:__call__] E=26 B=104300 s/sec=2151.67 tok/sec=97301.92 u/sec=23.47 ppl=6.606225 [2023-12-17:12:36:37:INFO:sockeye.training:__call__] E=26 B=104350 s/sec=2012.40 tok/sec=99975.00 u/sec=23.16 ppl=6.620446 [2023-12-17:12:36:39:INFO:sockeye.training:__call__] E=26 B=104400 s/sec=2087.68 tok/sec=101405.82 u/sec=22.81 ppl=6.630073 [2023-12-17:12:36:41:INFO:sockeye.training:__call__] E=26 B=104450 s/sec=2137.62 tok/sec=100467.97 u/sec=23.19 ppl=6.639066 [2023-12-17:12:36:43:INFO:sockeye.training:__call__] E=26 B=104500 s/sec=2254.81 tok/sec=95385.40 u/sec=23.72 ppl=6.665162 [2023-12-17:12:36:46:INFO:sockeye.training:__call__] E=26 B=104550 s/sec=1908.44 tok/sec=102105.36 u/sec=22.85 ppl=6.673099 [2023-12-17:12:36:48:INFO:sockeye.training:__call__] E=26 B=104600 s/sec=2045.95 tok/sec=98921.28 u/sec=23.29 ppl=6.676880 [2023-12-17:12:36:50:INFO:sockeye.training:__call__] E=26 B=104650 s/sec=2038.35 tok/sec=96804.29 u/sec=23.81 ppl=6.695172 [2023-12-17:12:36:52:INFO:sockeye.training:__call__] E=26 B=104700 s/sec=2026.31 tok/sec=99974.80 u/sec=23.28 ppl=6.707066 [2023-12-17:12:36:54:INFO:sockeye.training:__call__] E=26 B=104750 s/sec=1918.26 tok/sec=101591.52 u/sec=22.66 ppl=6.713886 [2023-12-17:12:36:56:INFO:sockeye.training:__call__] E=26 B=104800 s/sec=2131.00 tok/sec=99481.90 u/sec=23.57 ppl=6.728059 [2023-12-17:12:36:58:INFO:sockeye.training:__call__] E=26 B=104850 s/sec=2021.15 tok/sec=99919.83 u/sec=23.39 ppl=6.736957 [2023-12-17:12:37:00:INFO:sockeye.training:__call__] E=26 B=104900 s/sec=2407.58 tok/sec=93160.49 u/sec=24.31 ppl=6.759378 [2023-12-17:12:37:03:INFO:sockeye.training:__call__] E=26 B=104950 s/sec=1922.58 tok/sec=98940.56 u/sec=23.75 ppl=6.764101 [2023-12-17:12:37:05:INFO:sockeye.training:__call__] E=26 B=105000 s/sec=2208.45 tok/sec=97050.45 u/sec=23.72 ppl=6.780808 [2023-12-17:12:37:07:INFO:sockeye.training:__call__] E=26 B=105050 s/sec=2288.68 tok/sec=97931.23 u/sec=23.53 ppl=6.790344 [2023-12-17:12:37:09:INFO:sockeye.training:__call__] E=26 B=105100 s/sec=2065.74 tok/sec=96976.25 u/sec=23.65 ppl=6.801885 [2023-12-17:12:37:11:INFO:sockeye.training:__call__] E=26 B=105150 s/sec=2060.78 tok/sec=98499.11 u/sec=23.33 ppl=6.810818 [2023-12-17:12:37:13:INFO:sockeye.training:__call__] E=26 B=105200 s/sec=2101.22 tok/sec=98571.70 u/sec=23.20 ppl=6.821416 [2023-12-17:12:37:15:INFO:sockeye.training:__call__] E=26 B=105250 s/sec=2159.59 tok/sec=97240.86 u/sec=23.11 ppl=6.834068 [2023-12-17:12:37:17:INFO:sockeye.training:__call__] E=26 B=105300 s/sec=1977.41 tok/sec=99981.58 u/sec=23.14 ppl=6.838683 [2023-12-17:12:37:20:INFO:sockeye.training:__call__] E=26 B=105350 s/sec=1914.81 tok/sec=101084.36 u/sec=22.97 ppl=6.845588 [2023-12-17:12:37:22:INFO:sockeye.training:__call__] E=26 B=105400 s/sec=2114.55 tok/sec=100376.62 u/sec=23.06 ppl=6.854170 [2023-12-17:12:37:24:INFO:sockeye.training:__call__] E=26 B=105450 s/sec=2250.42 tok/sec=99447.27 u/sec=23.10 ppl=6.862777 [2023-12-17:12:37:26:INFO:sockeye.training:__call__] E=26 B=105500 s/sec=1965.07 tok/sec=100735.69 u/sec=23.09 ppl=6.869077 [2023-12-17:12:37:28:INFO:sockeye.training:__call__] E=26 B=105550 s/sec=1921.09 tok/sec=101172.37 u/sec=23.18 ppl=6.875622 [2023-12-17:12:37:30:INFO:sockeye.training:__call__] E=26 B=105600 s/sec=2039.91 tok/sec=99616.42 u/sec=23.31 ppl=6.882252 [2023-12-17:12:37:33:INFO:sockeye.training:__call__] E=26 B=105650 s/sec=1982.80 tok/sec=103237.18 u/sec=22.82 ppl=6.886545 [2023-12-17:12:37:35:INFO:sockeye.training:__call__] E=26 B=105700 s/sec=2058.42 tok/sec=99691.00 u/sec=23.10 ppl=6.895727 [2023-12-17:12:37:37:INFO:sockeye.training:__call__] E=26 B=105750 s/sec=1965.95 tok/sec=101197.13 u/sec=22.71 ppl=6.902095 [2023-12-17:12:37:39:INFO:sockeye.training:__call__] E=26 B=105800 s/sec=1865.60 tok/sec=103108.97 u/sec=22.68 ppl=6.905478 [2023-12-17:12:37:41:INFO:sockeye.training:__call__] E=26 B=105850 s/sec=2256.60 tok/sec=95854.03 u/sec=23.86 ppl=6.913436 [2023-12-17:12:37:43:INFO:sockeye.training:__call__] E=26 B=105900 s/sec=2167.58 tok/sec=97884.85 u/sec=23.36 ppl=6.919695 [2023-12-17:12:37:46:INFO:sockeye.training:__call__] E=26 B=105950 s/sec=1958.92 tok/sec=101878.52 u/sec=23.23 ppl=6.924835 [2023-12-17:12:37:48:INFO:sockeye.training:__call__] E=26 B=106000 s/sec=2098.31 tok/sec=93726.90 u/sec=22.57 ppl=6.936285 [2023-12-17:12:37:50:INFO:sockeye.training:__call__] E=26 B=106050 s/sec=2285.62 tok/sec=94614.19 u/sec=23.73 ppl=6.946259 [2023-12-17:12:37:52:INFO:sockeye.training:__call__] E=26 B=106100 s/sec=2060.25 tok/sec=99701.12 u/sec=23.41 ppl=6.953595 [2023-12-17:12:37:54:INFO:sockeye.training:__call__] E=26 B=106150 s/sec=1999.51 tok/sec=100893.88 u/sec=23.14 ppl=6.960082 [2023-12-17:12:37:56:INFO:sockeye.training:__call__] E=26 B=106200 s/sec=2191.46 tok/sec=97193.91 u/sec=23.57 ppl=6.968863 [2023-12-17:12:37:59:INFO:sockeye.training:__call__] E=26 B=106250 s/sec=2030.25 tok/sec=99034.43 u/sec=23.33 ppl=6.973890 [2023-12-17:12:38:01:INFO:sockeye.training:__call__] E=26 B=106300 s/sec=2222.51 tok/sec=97036.12 u/sec=23.11 ppl=6.985160 [2023-12-17:12:38:03:INFO:sockeye.training:__call__] E=26 B=106350 s/sec=2191.88 tok/sec=95530.38 u/sec=24.16 ppl=6.992612 [2023-12-17:12:38:05:INFO:sockeye.training:__call__] E=26 B=106400 s/sec=2008.46 tok/sec=100045.87 u/sec=23.12 ppl=6.998074 [2023-12-17:12:38:07:INFO:sockeye.training:__call__] E=26 B=106450 s/sec=2029.39 tok/sec=105825.86 u/sec=21.14 ppl=7.006570 [2023-12-17:12:38:09:INFO:sockeye.training:__call__] E=26 B=106500 s/sec=2133.10 tok/sec=98802.54 u/sec=23.35 ppl=7.014931 [2023-12-17:12:38:12:INFO:sockeye.training:__call__] E=26 B=106550 s/sec=2037.15 tok/sec=98784.77 u/sec=23.71 ppl=7.021293 [2023-12-17:12:38:14:INFO:sockeye.training:__call__] E=26 B=106600 s/sec=2292.34 tok/sec=98057.89 u/sec=23.56 ppl=7.030231 [2023-12-17:12:38:16:INFO:sockeye.training:__call__] E=26 B=106650 s/sec=2078.66 tok/sec=97663.21 u/sec=23.58 ppl=7.038260 [2023-12-17:12:38:18:INFO:sockeye.training:__call__] E=26 B=106700 s/sec=2143.07 tok/sec=98596.24 u/sec=23.33 ppl=7.044373 [2023-12-17:12:38:20:INFO:sockeye.training:__call__] E=26 B=106750 s/sec=2027.72 tok/sec=101060.21 u/sec=23.13 ppl=7.049528 [2023-12-17:12:38:22:INFO:sockeye.training:__call__] E=26 B=106800 s/sec=2052.87 tok/sec=98143.78 u/sec=23.67 ppl=7.055532 [2023-12-17:12:38:24:INFO:sockeye.training:__call__] E=26 B=106850 s/sec=2103.16 tok/sec=96580.72 u/sec=23.39 ppl=7.063876 [2023-12-17:12:38:26:INFO:sockeye.training:__call__] E=26 B=106900 s/sec=2080.91 tok/sec=99404.86 u/sec=23.39 ppl=7.071274 [2023-12-17:12:38:29:INFO:sockeye.training:__call__] E=26 B=106950 s/sec=2073.56 tok/sec=100468.81 u/sec=22.90 ppl=7.078738 [2023-12-17:12:38:31:INFO:sockeye.training:__call__] E=26 B=107000 s/sec=2024.11 tok/sec=96923.89 u/sec=22.79 ppl=7.084885 [2023-12-17:12:38:33:INFO:sockeye.training:__call__] E=26 B=107050 s/sec=2047.57 tok/sec=97550.64 u/sec=22.89 ppl=7.090173 [2023-12-17:12:38:35:INFO:sockeye.training:__call__] E=26 B=107100 s/sec=2058.29 tok/sec=99241.52 u/sec=23.10 ppl=7.096807 [2023-12-17:12:38:37:INFO:sockeye.training:__call__] E=26 B=107150 s/sec=2251.85 tok/sec=95059.62 u/sec=23.89 ppl=7.104901 [2023-12-17:12:38:39:INFO:sockeye.training:__call__] E=26 B=107200 s/sec=1774.61 tok/sec=102934.87 u/sec=22.77 ppl=7.107197 [2023-12-17:12:38:42:INFO:sockeye.training:__call__] E=26 B=107250 s/sec=2054.45 tok/sec=100382.98 u/sec=23.43 ppl=7.113836 [2023-12-17:12:38:44:INFO:sockeye.training:__call__] E=26 B=107300 s/sec=2335.81 tok/sec=94448.10 u/sec=24.05 ppl=7.121506 [2023-12-17:12:38:46:INFO:sockeye.training:__call__] E=26 B=107350 s/sec=1746.89 tok/sec=105848.58 u/sec=22.47 ppl=7.127065 [2023-12-17:12:38:48:INFO:sockeye.training:__call__] E=26 B=107400 s/sec=2139.57 tok/sec=99156.04 u/sec=23.46 ppl=7.135621 [2023-12-17:12:38:50:INFO:sockeye.training:__call__] E=26 B=107450 s/sec=2186.93 tok/sec=96632.73 u/sec=23.61 ppl=7.143041 [2023-12-17:12:38:52:INFO:sockeye.training:__call__] E=26 B=107500 s/sec=1617.85 tok/sec=103855.23 u/sec=22.22 ppl=7.145230 [2023-12-17:12:38:55:INFO:sockeye.training:__call__] E=26 B=107550 s/sec=2248.35 tok/sec=98480.73 u/sec=23.27 ppl=7.152001 [2023-12-17:12:38:57:INFO:sockeye.training:__call__] E=26 B=107600 s/sec=2085.55 tok/sec=99189.21 u/sec=23.11 ppl=7.157626 [2023-12-17:12:38:59:INFO:sockeye.training:__call__] E=26 B=107650 s/sec=1917.70 tok/sec=100937.70 u/sec=22.92 ppl=7.161759 [2023-12-17:12:39:01:INFO:sockeye.training:__call__] E=26 B=107700 s/sec=2233.16 tok/sec=100021.48 u/sec=23.34 ppl=7.166726 [2023-12-17:12:39:03:INFO:sockeye.training:__call__] E=26 B=107750 s/sec=2081.95 tok/sec=97558.55 u/sec=23.19 ppl=7.171582 [2023-12-17:12:39:05:INFO:sockeye.training:__call__] E=26 B=107800 s/sec=1854.66 tok/sec=104672.51 u/sec=22.64 ppl=7.175567 [2023-12-17:12:39:08:INFO:sockeye.training:__call__] E=26 B=107850 s/sec=1998.32 tok/sec=96003.75 u/sec=21.95 ppl=7.181196 [2023-12-17:12:39:10:INFO:sockeye.training:__call__] E=27 B=107900 s/sec=1998.63 tok/sec=92764.63 u/sec=22.11 ppl=7.175098 [2023-12-17:12:39:12:INFO:sockeye.training:__call__] E=27 B=107950 s/sec=1912.55 tok/sec=101427.48 u/sec=22.86 ppl=7.163458 [2023-12-17:12:39:14:INFO:sockeye.training:__call__] E=27 B=108000 s/sec=1954.15 tok/sec=97217.88 u/sec=22.83 ppl=7.153931 [2023-12-17:12:39:14:INFO:sockeye.training:_create_checkpoint] Checkpoint [27] Updates=108000 Epoch=27 Samples=9624648 Time-cost=172.676 Updates/sec=23.165 [2023-12-17:12:39:14:INFO:sockeye.training:_create_checkpoint] Checkpoint [27] Train-ppl=7.153931 [2023-12-17:12:39:17:INFO:sockeye.training:_evaluate] Checkpoint [27] Validation-ppl=30.596437 Validation-bleu=0.165977 Validation-chrf=31.435920 Validation-rouge1=0.169933 Validation-rouge2=0.082795 Validation-rougel=0.163457 Validation-length-ratio-mse=1.255662 Validation-ter=93.809327 Validation-avg-sec-per-sent=0.003425 Validation-decode-walltime=1.712455 [2023-12-17:12:39:17:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 3 checkpoints, best so far: 32.780032 [2023-12-17:12:39:17:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:12:39:17:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [27] Learning-rate=0.000162 [2023-12-17:12:39:18:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:39:18:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=3, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:39:18:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00027" [2023-12-17:12:39:20:INFO:sockeye.training:__call__] E=27 B=108050 s/sec=816.12 tok/sec=36088.84 u/sec=8.76 ppl=6.420340 [2023-12-17:12:39:22:INFO:sockeye.training:__call__] E=27 B=108100 s/sec=2018.84 tok/sec=99012.46 u/sec=23.19 ppl=6.417119 [2023-12-17:12:39:24:INFO:sockeye.training:__call__] E=27 B=108150 s/sec=2333.31 tok/sec=92880.65 u/sec=23.52 ppl=6.472234 [2023-12-17:12:39:27:INFO:sockeye.training:__call__] E=27 B=108200 s/sec=2151.54 tok/sec=96009.59 u/sec=22.79 ppl=6.476217 [2023-12-17:12:39:29:INFO:sockeye.training:__call__] E=27 B=108250 s/sec=1690.06 tok/sec=105431.07 u/sec=22.47 ppl=6.445113 [2023-12-17:12:39:31:INFO:sockeye.training:__call__] E=27 B=108300 s/sec=1992.02 tok/sec=99593.33 u/sec=23.18 ppl=6.448210 [2023-12-17:12:39:33:INFO:sockeye.training:__call__] E=27 B=108350 s/sec=2257.92 tok/sec=94168.51 u/sec=23.33 ppl=6.476064 [2023-12-17:12:39:35:INFO:sockeye.training:__call__] E=27 B=108400 s/sec=2057.13 tok/sec=101587.95 u/sec=22.92 ppl=6.467580 [2023-12-17:12:39:37:INFO:sockeye.training:__call__] E=27 B=108450 s/sec=2041.83 tok/sec=97344.16 u/sec=22.55 ppl=6.478141 [2023-12-17:12:39:40:INFO:sockeye.training:__call__] E=27 B=108500 s/sec=1785.19 tok/sec=97834.22 u/sec=22.86 ppl=6.476965 [2023-12-17:12:39:42:INFO:sockeye.training:__call__] E=27 B=108550 s/sec=2148.47 tok/sec=93963.73 u/sec=23.39 ppl=6.491218 [2023-12-17:12:39:44:INFO:sockeye.training:__call__] E=27 B=108600 s/sec=2004.76 tok/sec=98739.01 u/sec=23.08 ppl=6.502368 [2023-12-17:12:39:46:INFO:sockeye.training:__call__] E=27 B=108650 s/sec=1887.63 tok/sec=102785.32 u/sec=22.60 ppl=6.505378 [2023-12-17:12:39:48:INFO:sockeye.training:__call__] E=27 B=108700 s/sec=2053.16 tok/sec=99244.89 u/sec=23.55 ppl=6.513610 [2023-12-17:12:39:50:INFO:sockeye.training:__call__] E=27 B=108750 s/sec=2115.33 tok/sec=99499.30 u/sec=23.40 ppl=6.522736 [2023-12-17:12:39:53:INFO:sockeye.training:__call__] E=27 B=108800 s/sec=2016.43 tok/sec=98524.11 u/sec=22.23 ppl=6.531868 [2023-12-17:12:39:55:INFO:sockeye.training:__call__] E=27 B=108850 s/sec=2041.29 tok/sec=98731.08 u/sec=23.41 ppl=6.541381 [2023-12-17:12:39:57:INFO:sockeye.training:__call__] E=27 B=108900 s/sec=2024.48 tok/sec=100521.01 u/sec=23.13 ppl=6.550411 [2023-12-17:12:39:59:INFO:sockeye.training:__call__] E=27 B=108950 s/sec=2261.42 tok/sec=94664.67 u/sec=24.00 ppl=6.566480 [2023-12-17:12:40:01:INFO:sockeye.training:__call__] E=27 B=109000 s/sec=1955.61 tok/sec=99631.65 u/sec=23.33 ppl=6.574619 [2023-12-17:12:40:03:INFO:sockeye.training:__call__] E=27 B=109050 s/sec=2172.78 tok/sec=94305.52 u/sec=23.29 ppl=6.587373 [2023-12-17:12:40:06:INFO:sockeye.training:__call__] E=27 B=109100 s/sec=1983.52 tok/sec=101064.23 u/sec=22.87 ppl=6.597465 [2023-12-17:12:40:08:INFO:sockeye.training:__call__] E=27 B=109150 s/sec=1980.19 tok/sec=100647.24 u/sec=22.54 ppl=6.605166 [2023-12-17:12:40:10:INFO:sockeye.training:__call__] E=27 B=109200 s/sec=2018.38 tok/sec=98975.40 u/sec=23.36 ppl=6.617458 [2023-12-17:12:40:12:INFO:sockeye.training:__call__] E=27 B=109250 s/sec=2032.27 tok/sec=99523.32 u/sec=22.68 ppl=6.624014 [2023-12-17:12:40:14:INFO:sockeye.training:__call__] E=27 B=109300 s/sec=2255.01 tok/sec=96500.70 u/sec=24.01 ppl=6.637204 [2023-12-17:12:40:16:INFO:sockeye.training:__call__] E=27 B=109350 s/sec=2086.20 tok/sec=102559.24 u/sec=22.79 ppl=6.642878 [2023-12-17:12:40:19:INFO:sockeye.training:__call__] E=27 B=109400 s/sec=1895.62 tok/sec=102591.88 u/sec=23.01 ppl=6.652341 [2023-12-17:12:40:21:INFO:sockeye.training:__call__] E=27 B=109450 s/sec=2024.19 tok/sec=101363.74 u/sec=22.96 ppl=6.662435 [2023-12-17:12:40:23:INFO:sockeye.training:__call__] E=27 B=109500 s/sec=2298.73 tok/sec=95235.15 u/sec=23.55 ppl=6.674574 [2023-12-17:12:40:25:INFO:sockeye.training:__call__] E=27 B=109550 s/sec=2114.16 tok/sec=99985.60 u/sec=23.35 ppl=6.684108 [2023-12-17:12:40:27:INFO:sockeye.training:__call__] E=27 B=109600 s/sec=1999.85 tok/sec=99593.21 u/sec=22.68 ppl=6.694938 [2023-12-17:12:40:29:INFO:sockeye.training:__call__] E=27 B=109650 s/sec=1779.33 tok/sec=102697.05 u/sec=23.17 ppl=6.697865 [2023-12-17:12:40:31:INFO:sockeye.training:__call__] E=27 B=109700 s/sec=2187.03 tok/sec=96918.37 u/sec=23.41 ppl=6.709882 [2023-12-17:12:40:34:INFO:sockeye.training:__call__] E=27 B=109750 s/sec=1846.24 tok/sec=102271.54 u/sec=22.99 ppl=6.715477 [2023-12-17:12:40:36:INFO:sockeye.training:__call__] E=27 B=109800 s/sec=2246.85 tok/sec=97723.10 u/sec=23.40 ppl=6.723545 [2023-12-17:12:40:38:INFO:sockeye.training:__call__] E=27 B=109850 s/sec=1777.46 tok/sec=104411.41 u/sec=22.40 ppl=6.728690 [2023-12-17:12:40:40:INFO:sockeye.training:__call__] E=27 B=109900 s/sec=2307.67 tok/sec=98520.27 u/sec=23.34 ppl=6.740078 [2023-12-17:12:40:42:INFO:sockeye.training:__call__] E=27 B=109950 s/sec=2111.35 tok/sec=100279.98 u/sec=23.11 ppl=6.749360 [2023-12-17:12:40:44:INFO:sockeye.training:__call__] E=27 B=110000 s/sec=2075.30 tok/sec=97671.90 u/sec=23.71 ppl=6.756586 [2023-12-17:12:40:47:INFO:sockeye.training:__call__] E=27 B=110050 s/sec=1993.38 tok/sec=103082.10 u/sec=21.33 ppl=6.762470 [2023-12-17:12:40:49:INFO:sockeye.training:__call__] E=27 B=110100 s/sec=2227.36 tok/sec=98394.95 u/sec=23.52 ppl=6.771684 [2023-12-17:12:40:51:INFO:sockeye.training:__call__] E=27 B=110150 s/sec=1886.82 tok/sec=100035.01 u/sec=23.26 ppl=6.778199 [2023-12-17:12:40:53:INFO:sockeye.training:__call__] E=27 B=110200 s/sec=2044.01 tok/sec=99663.17 u/sec=22.03 ppl=6.787298 [2023-12-17:12:40:55:INFO:sockeye.training:__call__] E=27 B=110250 s/sec=2170.32 tok/sec=98835.40 u/sec=23.31 ppl=6.797847 [2023-12-17:12:40:58:INFO:sockeye.training:__call__] E=27 B=110300 s/sec=2155.65 tok/sec=99094.01 u/sec=22.95 ppl=6.804633 [2023-12-17:12:41:00:INFO:sockeye.training:__call__] E=27 B=110350 s/sec=2131.81 tok/sec=100545.49 u/sec=22.81 ppl=6.815006 [2023-12-17:12:41:02:INFO:sockeye.training:__call__] E=27 B=110400 s/sec=2259.36 tok/sec=95915.00 u/sec=23.65 ppl=6.824637 [2023-12-17:12:41:04:INFO:sockeye.training:__call__] E=27 B=110450 s/sec=2084.30 tok/sec=97077.44 u/sec=23.43 ppl=6.832094 [2023-12-17:12:41:06:INFO:sockeye.training:__call__] E=27 B=110500 s/sec=2168.19 tok/sec=98281.20 u/sec=23.32 ppl=6.839133 [2023-12-17:12:41:08:INFO:sockeye.training:__call__] E=27 B=110550 s/sec=2070.61 tok/sec=102328.30 u/sec=23.19 ppl=6.842747 [2023-12-17:12:41:11:INFO:sockeye.training:__call__] E=27 B=110600 s/sec=1827.65 tok/sec=103029.19 u/sec=23.12 ppl=6.846641 [2023-12-17:12:41:13:INFO:sockeye.training:__call__] E=27 B=110650 s/sec=2056.77 tok/sec=99327.83 u/sec=23.54 ppl=6.851993 [2023-12-17:12:41:15:INFO:sockeye.training:__call__] E=27 B=110700 s/sec=2126.88 tok/sec=98121.87 u/sec=23.49 ppl=6.858086 [2023-12-17:12:41:17:INFO:sockeye.training:__call__] E=27 B=110750 s/sec=2073.23 tok/sec=99247.33 u/sec=23.22 ppl=6.865256 [2023-12-17:12:41:19:INFO:sockeye.training:__call__] E=27 B=110800 s/sec=1825.89 tok/sec=102219.95 u/sec=23.24 ppl=6.869046 [2023-12-17:12:41:21:INFO:sockeye.training:__call__] E=27 B=110850 s/sec=1874.89 tok/sec=103403.15 u/sec=22.19 ppl=6.871588 [2023-12-17:12:41:24:INFO:sockeye.training:__call__] E=27 B=110900 s/sec=2174.96 tok/sec=99722.50 u/sec=23.12 ppl=6.879751 [2023-12-17:12:41:26:INFO:sockeye.training:__call__] E=27 B=110950 s/sec=1920.45 tok/sec=102716.52 u/sec=22.86 ppl=6.884210 [2023-12-17:12:41:28:INFO:sockeye.training:__call__] E=27 B=111000 s/sec=2193.62 tok/sec=90649.92 u/sec=23.16 ppl=6.891934 [2023-12-17:12:41:30:INFO:sockeye.training:__call__] E=27 B=111050 s/sec=1905.64 tok/sec=98735.77 u/sec=23.26 ppl=6.898137 [2023-12-17:12:41:32:INFO:sockeye.training:__call__] E=27 B=111100 s/sec=2304.97 tok/sec=97375.49 u/sec=23.93 ppl=6.905961 [2023-12-17:12:41:34:INFO:sockeye.training:__call__] E=27 B=111150 s/sec=2058.56 tok/sec=98990.69 u/sec=23.44 ppl=6.911199 [2023-12-17:12:41:36:INFO:sockeye.training:__call__] E=27 B=111200 s/sec=2318.28 tok/sec=97102.00 u/sec=23.71 ppl=6.918834 [2023-12-17:12:41:39:INFO:sockeye.training:__call__] E=27 B=111250 s/sec=1902.16 tok/sec=97903.48 u/sec=22.52 ppl=6.925702 [2023-12-17:12:41:41:INFO:sockeye.training:__call__] E=27 B=111300 s/sec=2059.10 tok/sec=100318.40 u/sec=23.15 ppl=6.931376 [2023-12-17:12:41:43:INFO:sockeye.training:__call__] E=27 B=111350 s/sec=2179.83 tok/sec=97922.48 u/sec=23.61 ppl=6.936628 [2023-12-17:12:41:45:INFO:sockeye.training:__call__] E=27 B=111400 s/sec=2118.56 tok/sec=96525.23 u/sec=22.67 ppl=6.943346 [2023-12-17:12:41:47:INFO:sockeye.training:__call__] E=27 B=111450 s/sec=2322.43 tok/sec=95734.46 u/sec=23.83 ppl=6.952102 [2023-12-17:12:41:49:INFO:sockeye.training:__call__] E=27 B=111500 s/sec=2130.89 tok/sec=98571.60 u/sec=23.57 ppl=6.958347 [2023-12-17:12:41:51:INFO:sockeye.training:__call__] E=27 B=111550 s/sec=1828.40 tok/sec=102593.70 u/sec=22.72 ppl=6.960448 [2023-12-17:12:41:54:INFO:sockeye.training:__call__] E=27 B=111600 s/sec=1948.87 tok/sec=99466.83 u/sec=23.25 ppl=6.964740 [2023-12-17:12:41:56:INFO:sockeye.training:__call__] E=27 B=111650 s/sec=2014.62 tok/sec=97902.05 u/sec=23.45 ppl=6.970451 [2023-12-17:12:41:58:INFO:sockeye.training:__call__] E=27 B=111700 s/sec=2133.64 tok/sec=96989.06 u/sec=23.81 ppl=6.975444 [2023-12-17:12:42:00:INFO:sockeye.training:__call__] E=27 B=111750 s/sec=2012.89 tok/sec=100016.77 u/sec=23.08 ppl=6.979198 [2023-12-17:12:42:02:INFO:sockeye.training:__call__] E=27 B=111800 s/sec=2211.07 tok/sec=97754.20 u/sec=23.54 ppl=6.985347 [2023-12-17:12:42:04:INFO:sockeye.training:__call__] E=27 B=111850 s/sec=2200.44 tok/sec=96954.71 u/sec=23.51 ppl=6.992944 [2023-12-17:12:42:07:INFO:sockeye.training:__call__] E=28 B=111900 s/sec=2126.44 tok/sec=91469.47 u/sec=22.56 ppl=6.987252 [2023-12-17:12:42:09:INFO:sockeye.training:__call__] E=28 B=111950 s/sec=2129.07 tok/sec=100676.53 u/sec=23.26 ppl=6.978002 [2023-12-17:12:42:11:INFO:sockeye.training:__call__] E=28 B=112000 s/sec=2115.91 tok/sec=98316.56 u/sec=23.49 ppl=6.968380 [2023-12-17:12:42:11:INFO:sockeye.training:_create_checkpoint] Checkpoint [28] Updates=112000 Epoch=28 Samples=9981944 Time-cost=172.861 Updates/sec=23.140 [2023-12-17:12:42:11:INFO:sockeye.training:_create_checkpoint] Checkpoint [28] Train-ppl=6.968380 [2023-12-17:12:42:13:INFO:sockeye.training:_evaluate] Checkpoint [28] Validation-ppl=31.389028 Validation-bleu=0.169759 Validation-chrf=31.224072 Validation-rouge1=0.167421 Validation-rouge2=0.083228 Validation-rougel=0.159678 Validation-length-ratio-mse=1.249069 Validation-ter=93.891870 Validation-avg-sec-per-sent=0.003415 Validation-decode-walltime=1.707390 [2023-12-17:12:42:13:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 4 checkpoints, best so far: 32.780032 [2023-12-17:12:42:13:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:12:42:13:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [28] Learning-rate=0.000162 [2023-12-17:12:42:14:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:42:14:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=4, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:42:14:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00028" [2023-12-17:12:42:17:INFO:sockeye.training:__call__] E=28 B=112050 s/sec=844.63 tok/sec=35000.50 u/sec=8.71 ppl=6.390150 [2023-12-17:12:42:19:INFO:sockeye.training:__call__] E=28 B=112100 s/sec=1801.69 tok/sec=102366.69 u/sec=21.70 ppl=6.284698 [2023-12-17:12:42:21:INFO:sockeye.training:__call__] E=28 B=112150 s/sec=1890.16 tok/sec=102819.06 u/sec=22.76 ppl=6.258243 [2023-12-17:12:42:23:INFO:sockeye.training:__call__] E=28 B=112200 s/sec=1931.45 tok/sec=100139.16 u/sec=23.13 ppl=6.252724 [2023-12-17:12:42:25:INFO:sockeye.training:__call__] E=28 B=112250 s/sec=2096.19 tok/sec=95729.84 u/sec=23.86 ppl=6.266773 [2023-12-17:12:42:27:INFO:sockeye.training:__call__] E=28 B=112300 s/sec=2027.68 tok/sec=99513.36 u/sec=23.38 ppl=6.284350 [2023-12-17:12:42:30:INFO:sockeye.training:__call__] E=28 B=112350 s/sec=2157.57 tok/sec=98923.04 u/sec=23.09 ppl=6.306647 [2023-12-17:12:42:32:INFO:sockeye.training:__call__] E=28 B=112400 s/sec=2118.26 tok/sec=96925.37 u/sec=23.64 ppl=6.329383 [2023-12-17:12:42:34:INFO:sockeye.training:__call__] E=28 B=112450 s/sec=2259.90 tok/sec=95928.60 u/sec=23.62 ppl=6.347202 [2023-12-17:12:42:36:INFO:sockeye.training:__call__] E=28 B=112500 s/sec=2045.34 tok/sec=100340.16 u/sec=23.16 ppl=6.354440 [2023-12-17:12:42:38:INFO:sockeye.training:__call__] E=28 B=112550 s/sec=2037.12 tok/sec=101464.72 u/sec=22.65 ppl=6.359123 [2023-12-17:12:42:40:INFO:sockeye.training:__call__] E=28 B=112600 s/sec=2002.57 tok/sec=101131.42 u/sec=22.88 ppl=6.363429 [2023-12-17:12:42:43:INFO:sockeye.training:__call__] E=28 B=112650 s/sec=2048.14 tok/sec=101587.86 u/sec=23.27 ppl=6.368392 [2023-12-17:12:42:45:INFO:sockeye.training:__call__] E=28 B=112700 s/sec=2168.93 tok/sec=98274.27 u/sec=23.62 ppl=6.379703 [2023-12-17:12:42:47:INFO:sockeye.training:__call__] E=28 B=112750 s/sec=1923.10 tok/sec=99956.70 u/sec=23.25 ppl=6.387251 [2023-12-17:12:42:49:INFO:sockeye.training:__call__] E=28 B=112800 s/sec=1914.28 tok/sec=100442.62 u/sec=23.05 ppl=6.397944 [2023-12-17:12:42:51:INFO:sockeye.training:__call__] E=28 B=112850 s/sec=2091.84 tok/sec=103321.32 u/sec=22.31 ppl=6.409883 [2023-12-17:12:42:53:INFO:sockeye.training:__call__] E=28 B=112900 s/sec=1979.45 tok/sec=102458.83 u/sec=23.08 ppl=6.419558 [2023-12-17:12:42:56:INFO:sockeye.training:__call__] E=28 B=112950 s/sec=2105.02 tok/sec=103596.21 u/sec=22.68 ppl=6.425047 [2023-12-17:12:42:58:INFO:sockeye.training:__call__] E=28 B=113000 s/sec=2365.92 tok/sec=93740.04 u/sec=24.16 ppl=6.437643 [2023-12-17:12:43:00:INFO:sockeye.training:__call__] E=28 B=113050 s/sec=1866.56 tok/sec=100482.09 u/sec=23.24 ppl=6.446336 [2023-12-17:12:43:02:INFO:sockeye.training:__call__] E=28 B=113100 s/sec=2030.03 tok/sec=99213.32 u/sec=23.07 ppl=6.456117 [2023-12-17:12:43:04:INFO:sockeye.training:__call__] E=28 B=113150 s/sec=1851.61 tok/sec=101520.50 u/sec=22.60 ppl=6.459899 [2023-12-17:12:43:06:INFO:sockeye.training:__call__] E=28 B=113200 s/sec=1735.59 tok/sec=93590.32 u/sec=21.65 ppl=6.464961 [2023-12-17:12:43:09:INFO:sockeye.training:__call__] E=28 B=113250 s/sec=2266.82 tok/sec=85315.38 u/sec=22.38 ppl=6.482585 [2023-12-17:12:43:11:INFO:sockeye.training:__call__] E=28 B=113300 s/sec=1858.69 tok/sec=98952.42 u/sec=21.92 ppl=6.490782 [2023-12-17:12:43:13:INFO:sockeye.training:__call__] E=28 B=113350 s/sec=2349.26 tok/sec=94526.75 u/sec=24.15 ppl=6.501230 [2023-12-17:12:43:15:INFO:sockeye.training:__call__] E=28 B=113400 s/sec=1715.40 tok/sec=102032.94 u/sec=22.11 ppl=6.506031 [2023-12-17:12:43:18:INFO:sockeye.training:__call__] E=28 B=113450 s/sec=2053.25 tok/sec=88665.90 u/sec=22.40 ppl=6.516619 [2023-12-17:12:43:20:INFO:sockeye.training:__call__] E=28 B=113500 s/sec=1948.44 tok/sec=92538.36 u/sec=22.02 ppl=6.525860 [2023-12-17:12:43:22:INFO:sockeye.training:__call__] E=28 B=113550 s/sec=1606.59 tok/sec=104591.53 u/sec=22.67 ppl=6.527780 [2023-12-17:12:43:24:INFO:sockeye.training:__call__] E=28 B=113600 s/sec=2177.98 tok/sec=96024.58 u/sec=23.27 ppl=6.538975 [2023-12-17:12:43:26:INFO:sockeye.training:__call__] E=28 B=113650 s/sec=2156.73 tok/sec=96098.47 u/sec=23.48 ppl=6.548520 [2023-12-17:12:43:28:INFO:sockeye.training:__call__] E=28 B=113700 s/sec=2087.48 tok/sec=100169.03 u/sec=23.42 ppl=6.554602 [2023-12-17:12:43:31:INFO:sockeye.training:__call__] E=28 B=113750 s/sec=2087.62 tok/sec=101334.55 u/sec=22.61 ppl=6.563680 [2023-12-17:12:43:33:INFO:sockeye.training:__call__] E=28 B=113800 s/sec=2179.58 tok/sec=98428.92 u/sec=23.37 ppl=6.574180 [2023-12-17:12:43:35:INFO:sockeye.training:__call__] E=28 B=113850 s/sec=2165.46 tok/sec=97193.32 u/sec=23.54 ppl=6.583793 [2023-12-17:12:43:37:INFO:sockeye.training:__call__] E=28 B=113900 s/sec=2236.86 tok/sec=96113.45 u/sec=23.58 ppl=6.591457 [2023-12-17:12:43:39:INFO:sockeye.training:__call__] E=28 B=113950 s/sec=1872.99 tok/sec=105455.93 u/sec=22.21 ppl=6.595672 [2023-12-17:12:43:42:INFO:sockeye.training:__call__] E=28 B=114000 s/sec=2065.23 tok/sec=98287.49 u/sec=22.72 ppl=6.602606 [2023-12-17:12:43:44:INFO:sockeye.training:__call__] E=28 B=114050 s/sec=1925.00 tok/sec=100615.14 u/sec=23.00 ppl=6.607504 [2023-12-17:12:43:46:INFO:sockeye.training:__call__] E=28 B=114100 s/sec=2164.52 tok/sec=95890.40 u/sec=24.24 ppl=6.612988 [2023-12-17:12:43:48:INFO:sockeye.training:__call__] E=28 B=114150 s/sec=1814.63 tok/sec=100264.51 u/sec=23.00 ppl=6.615848 [2023-12-17:12:43:50:INFO:sockeye.training:__call__] E=28 B=114200 s/sec=2022.42 tok/sec=101353.58 u/sec=22.73 ppl=6.622380 [2023-12-17:12:43:52:INFO:sockeye.training:__call__] E=28 B=114250 s/sec=1688.40 tok/sec=104151.98 u/sec=22.94 ppl=6.626060 [2023-12-17:12:43:54:INFO:sockeye.training:__call__] E=28 B=114300 s/sec=2207.46 tok/sec=99638.26 u/sec=23.07 ppl=6.634538 [2023-12-17:12:43:57:INFO:sockeye.training:__call__] E=28 B=114350 s/sec=2131.05 tok/sec=98013.06 u/sec=23.53 ppl=6.642004 [2023-12-17:12:43:59:INFO:sockeye.training:__call__] E=28 B=114400 s/sec=2245.23 tok/sec=98834.74 u/sec=23.35 ppl=6.651790 [2023-12-17:12:44:01:INFO:sockeye.training:__call__] E=28 B=114450 s/sec=2207.88 tok/sec=98533.11 u/sec=23.55 ppl=6.658938 [2023-12-17:12:44:03:INFO:sockeye.training:__call__] E=28 B=114500 s/sec=2042.32 tok/sec=99955.23 u/sec=23.12 ppl=6.665036 [2023-12-17:12:44:05:INFO:sockeye.training:__call__] E=28 B=114550 s/sec=2218.65 tok/sec=95520.73 u/sec=23.95 ppl=6.671182 [2023-12-17:12:44:07:INFO:sockeye.training:__call__] E=28 B=114600 s/sec=2071.19 tok/sec=99269.81 u/sec=22.99 ppl=6.677129 [2023-12-17:12:44:10:INFO:sockeye.training:__call__] E=28 B=114650 s/sec=1773.13 tok/sec=105402.58 u/sec=21.99 ppl=6.682421 [2023-12-17:12:44:12:INFO:sockeye.training:__call__] E=28 B=114700 s/sec=1995.69 tok/sec=97818.20 u/sec=22.93 ppl=6.689759 [2023-12-17:12:44:14:INFO:sockeye.training:__call__] E=28 B=114750 s/sec=2150.80 tok/sec=96367.87 u/sec=23.75 ppl=6.696263 [2023-12-17:12:44:16:INFO:sockeye.training:__call__] E=28 B=114800 s/sec=2097.16 tok/sec=99016.01 u/sec=23.41 ppl=6.701942 [2023-12-17:12:44:18:INFO:sockeye.training:__call__] E=28 B=114850 s/sec=2141.80 tok/sec=99338.95 u/sec=22.57 ppl=6.708252 [2023-12-17:12:44:20:INFO:sockeye.training:__call__] E=28 B=114900 s/sec=1921.55 tok/sec=100113.04 u/sec=23.14 ppl=6.713855 [2023-12-17:12:44:23:INFO:sockeye.training:__call__] E=28 B=114950 s/sec=2019.44 tok/sec=99279.11 u/sec=23.20 ppl=6.720485 [2023-12-17:12:44:25:INFO:sockeye.training:__call__] E=28 B=115000 s/sec=2204.58 tok/sec=97001.33 u/sec=24.00 ppl=6.727868 [2023-12-17:12:44:27:INFO:sockeye.training:__call__] E=28 B=115050 s/sec=2131.54 tok/sec=97657.75 u/sec=23.62 ppl=6.735243 [2023-12-17:12:44:29:INFO:sockeye.training:__call__] E=28 B=115100 s/sec=2099.42 tok/sec=100213.40 u/sec=22.98 ppl=6.742635 [2023-12-17:12:44:31:INFO:sockeye.training:__call__] E=28 B=115150 s/sec=2034.27 tok/sec=100993.68 u/sec=22.95 ppl=6.749894 [2023-12-17:12:44:33:INFO:sockeye.training:__call__] E=28 B=115200 s/sec=2012.92 tok/sec=99690.24 u/sec=23.52 ppl=6.754185 [2023-12-17:12:44:35:INFO:sockeye.training:__call__] E=28 B=115250 s/sec=2324.31 tok/sec=96413.61 u/sec=23.58 ppl=6.761500 [2023-12-17:12:44:38:INFO:sockeye.training:__call__] E=28 B=115300 s/sec=2096.07 tok/sec=99262.64 u/sec=22.90 ppl=6.766438 [2023-12-17:12:44:40:INFO:sockeye.training:__call__] E=28 B=115350 s/sec=1942.54 tok/sec=102738.57 u/sec=22.48 ppl=6.771331 [2023-12-17:12:44:42:INFO:sockeye.training:__call__] E=28 B=115400 s/sec=1965.22 tok/sec=100192.93 u/sec=23.13 ppl=6.775410 [2023-12-17:12:44:44:INFO:sockeye.training:__call__] E=28 B=115450 s/sec=1976.51 tok/sec=99509.29 u/sec=23.22 ppl=6.779908 [2023-12-17:12:44:46:INFO:sockeye.training:__call__] E=28 B=115500 s/sec=2188.35 tok/sec=98617.51 u/sec=23.34 ppl=6.784885 [2023-12-17:12:44:48:INFO:sockeye.training:__call__] E=28 B=115550 s/sec=1993.07 tok/sec=99394.70 u/sec=23.11 ppl=6.789380 [2023-12-17:12:44:50:INFO:sockeye.training:__call__] E=28 B=115600 s/sec=2113.27 tok/sec=98270.67 u/sec=23.34 ppl=6.795344 [2023-12-17:12:44:53:INFO:sockeye.training:__call__] E=28 B=115650 s/sec=2284.26 tok/sec=97251.84 u/sec=23.44 ppl=6.804172 [2023-12-17:12:44:55:INFO:sockeye.training:__call__] E=28 B=115700 s/sec=2223.50 tok/sec=96311.24 u/sec=23.80 ppl=6.809891 [2023-12-17:12:44:57:INFO:sockeye.training:__call__] E=28 B=115750 s/sec=2214.29 tok/sec=97504.17 u/sec=23.58 ppl=6.816283 [2023-12-17:12:44:59:INFO:sockeye.training:__call__] E=28 B=115800 s/sec=2028.09 tok/sec=101677.05 u/sec=23.00 ppl=6.821279 [2023-12-17:12:45:01:INFO:sockeye.training:__call__] E=28 B=115850 s/sec=2264.88 tok/sec=99294.89 u/sec=23.44 ppl=6.827749 [2023-12-17:12:45:03:INFO:sockeye.training:__call__] E=29 B=115900 s/sec=2005.45 tok/sec=91315.87 u/sec=23.00 ppl=6.820566 [2023-12-17:12:45:05:INFO:sockeye.training:__call__] E=29 B=115950 s/sec=1844.63 tok/sec=101584.10 u/sec=23.10 ppl=6.809538 [2023-12-17:12:45:08:INFO:sockeye.training:__call__] E=29 B=116000 s/sec=2129.01 tok/sec=99025.02 u/sec=23.34 ppl=6.800896 [2023-12-17:12:45:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [29] Updates=116000 Epoch=29 Samples=10337600 Time-cost=173.206 Updates/sec=23.094 [2023-12-17:12:45:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [29] Train-ppl=6.800896 [2023-12-17:12:45:10:INFO:sockeye.training:_evaluate] Checkpoint [29] Validation-ppl=32.048300 Validation-bleu=0.155628 Validation-chrf=30.123067 Validation-rouge1=0.158398 Validation-rouge2=0.075613 Validation-rougel=0.151485 Validation-length-ratio-mse=1.324581 Validation-ter=97.688816 Validation-avg-sec-per-sent=0.003493 Validation-decode-walltime=1.746609 [2023-12-17:12:45:10:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 5 checkpoints, best so far: 32.780032 [2023-12-17:12:45:10:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:12:45:10:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [29] Learning-rate=0.000162 [2023-12-17:12:45:11:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:45:11:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=5, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:45:11:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00029" [2023-12-17:12:45:13:INFO:sockeye.training:__call__] E=29 B=116050 s/sec=766.08 tok/sec=39830.68 u/sec=8.95 ppl=6.105870 [2023-12-17:12:45:15:INFO:sockeye.training:__call__] E=29 B=116100 s/sec=2052.58 tok/sec=101172.44 u/sec=22.99 ppl=6.113376 [2023-12-17:12:45:18:INFO:sockeye.training:__call__] E=29 B=116150 s/sec=2284.98 tok/sec=97613.59 u/sec=23.68 ppl=6.139344 [2023-12-17:12:45:20:INFO:sockeye.training:__call__] E=29 B=116200 s/sec=2000.07 tok/sec=100728.10 u/sec=22.65 ppl=6.152866 [2023-12-17:12:45:22:INFO:sockeye.training:__call__] E=29 B=116250 s/sec=2131.65 tok/sec=95314.52 u/sec=23.09 ppl=6.174623 [2023-12-17:12:45:24:INFO:sockeye.training:__call__] E=29 B=116300 s/sec=1906.23 tok/sec=101895.25 u/sec=22.44 ppl=6.182109 [2023-12-17:12:45:26:INFO:sockeye.training:__call__] E=29 B=116350 s/sec=2361.86 tok/sec=88985.61 u/sec=24.64 ppl=6.201161 [2023-12-17:12:45:28:INFO:sockeye.training:__call__] E=29 B=116400 s/sec=2129.90 tok/sec=97757.38 u/sec=23.48 ppl=6.226717 [2023-12-17:12:45:30:INFO:sockeye.training:__call__] E=29 B=116450 s/sec=1876.65 tok/sec=102489.09 u/sec=22.60 ppl=6.224577 [2023-12-17:12:45:33:INFO:sockeye.training:__call__] E=29 B=116500 s/sec=2098.13 tok/sec=97922.68 u/sec=23.17 ppl=6.227758 [2023-12-17:12:45:35:INFO:sockeye.training:__call__] E=29 B=116550 s/sec=2029.31 tok/sec=98983.69 u/sec=22.81 ppl=6.238108 [2023-12-17:12:45:37:INFO:sockeye.training:__call__] E=29 B=116600 s/sec=2138.57 tok/sec=91357.13 u/sec=23.78 ppl=6.246993 [2023-12-17:12:45:39:INFO:sockeye.training:__call__] E=29 B=116650 s/sec=2109.84 tok/sec=97872.85 u/sec=23.30 ppl=6.256195 [2023-12-17:12:45:41:INFO:sockeye.training:__call__] E=29 B=116700 s/sec=1939.39 tok/sec=100298.27 u/sec=22.61 ppl=6.262035 [2023-12-17:12:45:44:INFO:sockeye.training:__call__] E=29 B=116750 s/sec=1945.65 tok/sec=96830.79 u/sec=22.39 ppl=6.266781 [2023-12-17:12:45:46:INFO:sockeye.training:__call__] E=29 B=116800 s/sec=2149.75 tok/sec=93123.16 u/sec=23.61 ppl=6.283012 [2023-12-17:12:45:48:INFO:sockeye.training:__call__] E=29 B=116850 s/sec=2412.74 tok/sec=94732.50 u/sec=23.94 ppl=6.297564 [2023-12-17:12:45:50:INFO:sockeye.training:__call__] E=29 B=116900 s/sec=1895.69 tok/sec=101783.50 u/sec=23.10 ppl=6.306373 [2023-12-17:12:45:52:INFO:sockeye.training:__call__] E=29 B=116950 s/sec=2058.04 tok/sec=97937.08 u/sec=23.69 ppl=6.314245 [2023-12-17:12:45:54:INFO:sockeye.training:__call__] E=29 B=117000 s/sec=2413.69 tok/sec=95254.05 u/sec=24.06 ppl=6.329865 [2023-12-17:12:45:56:INFO:sockeye.training:__call__] E=29 B=117050 s/sec=2309.59 tok/sec=96610.21 u/sec=23.59 ppl=6.342876 [2023-12-17:12:45:58:INFO:sockeye.training:__call__] E=29 B=117100 s/sec=2338.93 tok/sec=94841.22 u/sec=23.89 ppl=6.353458 [2023-12-17:12:46:00:INFO:sockeye.training:__call__] E=29 B=117150 s/sec=1960.43 tok/sec=99001.73 u/sec=23.21 ppl=6.362545 [2023-12-17:12:46:03:INFO:sockeye.training:__call__] E=29 B=117200 s/sec=1909.58 tok/sec=101981.89 u/sec=23.04 ppl=6.367592 [2023-12-17:12:46:05:INFO:sockeye.training:__call__] E=29 B=117250 s/sec=2369.40 tok/sec=93850.02 u/sec=24.12 ppl=6.378014 [2023-12-17:12:46:07:INFO:sockeye.training:__call__] E=29 B=117300 s/sec=2045.14 tok/sec=96419.22 u/sec=23.54 ppl=6.388227 [2023-12-17:12:46:09:INFO:sockeye.training:__call__] E=29 B=117350 s/sec=1992.62 tok/sec=101066.16 u/sec=22.77 ppl=6.396400 [2023-12-17:12:46:11:INFO:sockeye.training:__call__] E=29 B=117400 s/sec=2158.50 tok/sec=94517.96 u/sec=23.75 ppl=6.404399 [2023-12-17:12:46:13:INFO:sockeye.training:__call__] E=29 B=117450 s/sec=2077.80 tok/sec=98450.94 u/sec=23.31 ppl=6.409936 [2023-12-17:12:46:15:INFO:sockeye.training:__call__] E=29 B=117500 s/sec=1787.22 tok/sec=102973.33 u/sec=22.80 ppl=6.411826 [2023-12-17:12:46:18:INFO:sockeye.training:__call__] E=29 B=117550 s/sec=1910.75 tok/sec=99373.66 u/sec=22.66 ppl=6.416108 [2023-12-17:12:46:20:INFO:sockeye.training:__call__] E=29 B=117600 s/sec=2149.36 tok/sec=96758.43 u/sec=23.40 ppl=6.425193 [2023-12-17:12:46:22:INFO:sockeye.training:__call__] E=29 B=117650 s/sec=1864.58 tok/sec=103076.57 u/sec=22.76 ppl=6.428604 [2023-12-17:12:46:24:INFO:sockeye.training:__call__] E=29 B=117700 s/sec=1994.46 tok/sec=101654.34 u/sec=23.13 ppl=6.434470 [2023-12-17:12:46:26:INFO:sockeye.training:__call__] E=29 B=117750 s/sec=2062.60 tok/sec=97980.85 u/sec=23.52 ppl=6.443377 [2023-12-17:12:46:28:INFO:sockeye.training:__call__] E=29 B=117800 s/sec=2028.55 tok/sec=99014.46 u/sec=23.79 ppl=6.450812 [2023-12-17:12:46:31:INFO:sockeye.training:__call__] E=29 B=117850 s/sec=1869.82 tok/sec=101903.55 u/sec=23.33 ppl=6.456471 [2023-12-17:12:46:33:INFO:sockeye.training:__call__] E=29 B=117900 s/sec=1857.92 tok/sec=104160.04 u/sec=22.77 ppl=6.461676 [2023-12-17:12:46:35:INFO:sockeye.training:__call__] E=29 B=117950 s/sec=1974.02 tok/sec=104245.07 u/sec=22.07 ppl=6.468176 [2023-12-17:12:46:37:INFO:sockeye.training:__call__] E=29 B=118000 s/sec=2128.87 tok/sec=99062.52 u/sec=23.02 ppl=6.474436 [2023-12-17:12:46:39:INFO:sockeye.training:__call__] E=29 B=118050 s/sec=1971.45 tok/sec=97919.25 u/sec=22.95 ppl=6.479503 [2023-12-17:12:46:42:INFO:sockeye.training:__call__] E=29 B=118100 s/sec=1934.18 tok/sec=101754.05 u/sec=22.98 ppl=6.486206 [2023-12-17:12:46:44:INFO:sockeye.training:__call__] E=29 B=118150 s/sec=2144.18 tok/sec=97849.14 u/sec=23.31 ppl=6.492030 [2023-12-17:12:46:46:INFO:sockeye.training:__call__] E=29 B=118200 s/sec=1809.76 tok/sec=103546.98 u/sec=22.62 ppl=6.495159 [2023-12-17:12:46:48:INFO:sockeye.training:__call__] E=29 B=118250 s/sec=2175.35 tok/sec=95776.67 u/sec=23.94 ppl=6.500696 [2023-12-17:12:46:50:INFO:sockeye.training:__call__] E=29 B=118300 s/sec=1859.53 tok/sec=101527.64 u/sec=22.88 ppl=6.504648 [2023-12-17:12:46:52:INFO:sockeye.training:__call__] E=29 B=118350 s/sec=1959.89 tok/sec=100698.06 u/sec=22.90 ppl=6.510634 [2023-12-17:12:46:55:INFO:sockeye.training:__call__] E=29 B=118400 s/sec=1990.97 tok/sec=103398.78 u/sec=22.83 ppl=6.516478 [2023-12-17:12:46:57:INFO:sockeye.training:__call__] E=29 B=118450 s/sec=1966.57 tok/sec=101701.90 u/sec=23.02 ppl=6.521527 [2023-12-17:12:46:59:INFO:sockeye.training:__call__] E=29 B=118500 s/sec=1970.19 tok/sec=101714.50 u/sec=22.97 ppl=6.526429 [2023-12-17:12:47:01:INFO:sockeye.training:__call__] E=29 B=118550 s/sec=2100.77 tok/sec=98887.42 u/sec=23.08 ppl=6.532284 [2023-12-17:12:47:03:INFO:sockeye.training:__call__] E=29 B=118600 s/sec=2231.35 tok/sec=96891.56 u/sec=23.40 ppl=6.537941 [2023-12-17:12:47:05:INFO:sockeye.training:__call__] E=29 B=118650 s/sec=2089.12 tok/sec=99322.62 u/sec=23.32 ppl=6.545142 [2023-12-17:12:47:08:INFO:sockeye.training:__call__] E=29 B=118700 s/sec=1882.24 tok/sec=104723.52 u/sec=22.20 ppl=6.549732 [2023-12-17:12:47:10:INFO:sockeye.training:__call__] E=29 B=118750 s/sec=2090.19 tok/sec=98762.22 u/sec=23.54 ppl=6.556588 [2023-12-17:12:47:12:INFO:sockeye.training:__call__] E=29 B=118800 s/sec=2286.60 tok/sec=99354.33 u/sec=23.09 ppl=6.564652 [2023-12-17:12:47:14:INFO:sockeye.training:__call__] E=29 B=118850 s/sec=2141.35 tok/sec=101548.50 u/sec=23.00 ppl=6.570187 [2023-12-17:12:47:16:INFO:sockeye.training:__call__] E=29 B=118900 s/sec=2061.37 tok/sec=99868.51 u/sec=23.26 ppl=6.575462 [2023-12-17:12:47:18:INFO:sockeye.training:__call__] E=29 B=118950 s/sec=2236.91 tok/sec=95623.95 u/sec=23.78 ppl=6.582277 [2023-12-17:12:47:20:INFO:sockeye.training:__call__] E=29 B=119000 s/sec=1968.39 tok/sec=98495.13 u/sec=23.57 ppl=6.586427 [2023-12-17:12:47:23:INFO:sockeye.training:__call__] E=29 B=119050 s/sec=2193.92 tok/sec=96234.93 u/sec=23.24 ppl=6.592900 [2023-12-17:12:47:25:INFO:sockeye.training:__call__] E=29 B=119100 s/sec=2099.46 tok/sec=96473.26 u/sec=22.70 ppl=6.599729 [2023-12-17:12:47:27:INFO:sockeye.training:__call__] E=29 B=119150 s/sec=2213.25 tok/sec=97047.09 u/sec=23.85 ppl=6.607356 [2023-12-17:12:47:29:INFO:sockeye.training:__call__] E=29 B=119200 s/sec=2196.48 tok/sec=99167.96 u/sec=23.19 ppl=6.613992 [2023-12-17:12:47:31:INFO:sockeye.training:__call__] E=29 B=119250 s/sec=2183.07 tok/sec=98483.13 u/sec=23.57 ppl=6.620188 [2023-12-17:12:47:33:INFO:sockeye.training:__call__] E=29 B=119300 s/sec=2081.13 tok/sec=98738.95 u/sec=23.14 ppl=6.625070 [2023-12-17:12:47:35:INFO:sockeye.training:__call__] E=29 B=119350 s/sec=2053.06 tok/sec=99758.85 u/sec=23.67 ppl=6.630705 [2023-12-17:12:47:38:INFO:sockeye.training:__call__] E=29 B=119400 s/sec=2029.77 tok/sec=100971.88 u/sec=23.07 ppl=6.635158 [2023-12-17:12:47:40:INFO:sockeye.training:__call__] E=29 B=119450 s/sec=2415.68 tok/sec=96777.88 u/sec=23.52 ppl=6.642558 [2023-12-17:12:47:42:INFO:sockeye.training:__call__] E=29 B=119500 s/sec=1948.18 tok/sec=101924.76 u/sec=22.51 ppl=6.645057 [2023-12-17:12:47:44:INFO:sockeye.training:__call__] E=29 B=119550 s/sec=2224.10 tok/sec=98593.12 u/sec=23.36 ppl=6.651982 [2023-12-17:12:47:46:INFO:sockeye.training:__call__] E=29 B=119600 s/sec=2173.54 tok/sec=95756.51 u/sec=23.58 ppl=6.658127 [2023-12-17:12:47:48:INFO:sockeye.training:__call__] E=29 B=119650 s/sec=2069.03 tok/sec=103182.31 u/sec=22.73 ppl=6.663028 [2023-12-17:12:47:51:INFO:sockeye.training:__call__] E=29 B=119700 s/sec=1922.52 tok/sec=101537.63 u/sec=22.46 ppl=6.667300 [2023-12-17:12:47:53:INFO:sockeye.training:__call__] E=29 B=119750 s/sec=2094.47 tok/sec=98063.36 u/sec=23.54 ppl=6.671864 [2023-12-17:12:47:55:INFO:sockeye.training:__call__] E=29 B=119800 s/sec=1988.37 tok/sec=101247.70 u/sec=23.14 ppl=6.675618 [2023-12-17:12:47:57:INFO:sockeye.training:__call__] E=29 B=119850 s/sec=1962.41 tok/sec=108560.96 u/sec=21.71 ppl=6.679838 [2023-12-17:12:48:00:INFO:sockeye.training:__call__] E=30 B=119900 s/sec=1871.32 tok/sec=98461.33 u/sec=21.70 ppl=6.669639 [2023-12-17:12:48:02:INFO:sockeye.training:__call__] E=30 B=119950 s/sec=2121.03 tok/sec=94808.63 u/sec=23.18 ppl=6.661524 [2023-12-17:12:48:04:INFO:sockeye.training:__call__] E=30 B=120000 s/sec=1862.97 tok/sec=103146.50 u/sec=22.48 ppl=6.652088 [2023-12-17:12:48:04:INFO:sockeye.training:_create_checkpoint] Checkpoint [30] Updates=120000 Epoch=30 Samples=10694232 Time-cost=172.880 Updates/sec=23.137 [2023-12-17:12:48:04:INFO:sockeye.training:_create_checkpoint] Checkpoint [30] Train-ppl=6.652088 [2023-12-17:12:48:07:INFO:sockeye.training:_evaluate] Checkpoint [30] Validation-ppl=33.073982 Validation-bleu=0.169554 Validation-chrf=31.937972 Validation-rouge1=0.167111 Validation-rouge2=0.081628 Validation-rougel=0.159600 Validation-length-ratio-mse=1.325885 Validation-ter=97.977714 Validation-avg-sec-per-sent=0.003596 Validation-decode-walltime=1.798089 [2023-12-17:12:48:07:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 6 checkpoints, best so far: 32.780032 [2023-12-17:12:48:07:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:12:48:07:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [30] Learning-rate=0.000162 [2023-12-17:12:48:07:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:48:07:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=6, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:48:08:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00030" [2023-12-17:12:48:10:INFO:sockeye.training:__call__] E=30 B=120050 s/sec=764.25 tok/sec=35218.20 u/sec=8.54 ppl=6.025048 [2023-12-17:12:48:12:INFO:sockeye.training:__call__] E=30 B=120100 s/sec=1848.99 tok/sec=104541.08 u/sec=22.93 ppl=6.004863 [2023-12-17:12:48:14:INFO:sockeye.training:__call__] E=30 B=120150 s/sec=1948.24 tok/sec=99164.45 u/sec=23.11 ppl=6.005783 [2023-12-17:12:48:16:INFO:sockeye.training:__call__] E=30 B=120200 s/sec=2079.07 tok/sec=98632.89 u/sec=23.29 ppl=6.038519 [2023-12-17:12:48:18:INFO:sockeye.training:__call__] E=30 B=120250 s/sec=2072.86 tok/sec=99076.12 u/sec=23.51 ppl=6.044085 [2023-12-17:12:48:21:INFO:sockeye.training:__call__] E=30 B=120300 s/sec=2069.10 tok/sec=101315.24 u/sec=23.30 ppl=6.053422 [2023-12-17:12:48:23:INFO:sockeye.training:__call__] E=30 B=120350 s/sec=2340.78 tok/sec=95698.38 u/sec=23.75 ppl=6.073085 [2023-12-17:12:48:25:INFO:sockeye.training:__call__] E=30 B=120400 s/sec=2046.26 tok/sec=98220.31 u/sec=23.38 ppl=6.072530 [2023-12-17:12:48:27:INFO:sockeye.training:__call__] E=30 B=120450 s/sec=2199.50 tok/sec=97431.30 u/sec=23.74 ppl=6.085159 [2023-12-17:12:48:29:INFO:sockeye.training:__call__] E=30 B=120500 s/sec=2428.55 tok/sec=95159.57 u/sec=23.83 ppl=6.110693 [2023-12-17:12:48:31:INFO:sockeye.training:__call__] E=30 B=120550 s/sec=1821.63 tok/sec=103767.10 u/sec=22.86 ppl=6.108706 [2023-12-17:12:48:33:INFO:sockeye.training:__call__] E=30 B=120600 s/sec=2214.60 tok/sec=97322.45 u/sec=23.46 ppl=6.118779 [2023-12-17:12:48:35:INFO:sockeye.training:__call__] E=30 B=120650 s/sec=2055.65 tok/sec=98581.54 u/sec=23.32 ppl=6.130791 [2023-12-17:12:48:38:INFO:sockeye.training:__call__] E=30 B=120700 s/sec=2136.17 tok/sec=98046.80 u/sec=23.38 ppl=6.143848 [2023-12-17:12:48:40:INFO:sockeye.training:__call__] E=30 B=120750 s/sec=1988.32 tok/sec=99481.41 u/sec=22.72 ppl=6.153911 [2023-12-17:12:48:42:INFO:sockeye.training:__call__] E=30 B=120800 s/sec=2079.79 tok/sec=100007.21 u/sec=23.09 ppl=6.163001 [2023-12-17:12:48:44:INFO:sockeye.training:__call__] E=30 B=120850 s/sec=2072.72 tok/sec=100598.17 u/sec=22.77 ppl=6.167255 [2023-12-17:12:48:46:INFO:sockeye.training:__call__] E=30 B=120900 s/sec=2095.21 tok/sec=98218.85 u/sec=23.55 ppl=6.174272 [2023-12-17:12:48:48:INFO:sockeye.training:__call__] E=30 B=120950 s/sec=1926.37 tok/sec=102079.16 u/sec=22.76 ppl=6.181559 [2023-12-17:12:48:51:INFO:sockeye.training:__call__] E=30 B=121000 s/sec=2063.41 tok/sec=99043.53 u/sec=23.41 ppl=6.192053 [2023-12-17:12:48:53:INFO:sockeye.training:__call__] E=30 B=121050 s/sec=2160.91 tok/sec=97738.60 u/sec=23.74 ppl=6.203075 [2023-12-17:12:48:55:INFO:sockeye.training:__call__] E=30 B=121100 s/sec=2326.40 tok/sec=92371.08 u/sec=23.26 ppl=6.212951 [2023-12-17:12:48:57:INFO:sockeye.training:__call__] E=30 B=121150 s/sec=2391.51 tok/sec=94073.88 u/sec=23.84 ppl=6.226667 [2023-12-17:12:48:59:INFO:sockeye.training:__call__] E=30 B=121200 s/sec=1821.10 tok/sec=101951.86 u/sec=23.13 ppl=6.233826 [2023-12-17:12:49:01:INFO:sockeye.training:__call__] E=30 B=121250 s/sec=2092.97 tok/sec=101407.64 u/sec=23.07 ppl=6.242899 [2023-12-17:12:49:04:INFO:sockeye.training:__call__] E=30 B=121300 s/sec=2038.34 tok/sec=105579.23 u/sec=21.59 ppl=6.253408 [2023-12-17:12:49:06:INFO:sockeye.training:__call__] E=30 B=121350 s/sec=2020.87 tok/sec=98509.34 u/sec=23.09 ppl=6.263427 [2023-12-17:12:49:08:INFO:sockeye.training:__call__] E=30 B=121400 s/sec=2244.90 tok/sec=99395.90 u/sec=23.08 ppl=6.271587 [2023-12-17:12:49:10:INFO:sockeye.training:__call__] E=30 B=121450 s/sec=1863.74 tok/sec=101197.29 u/sec=22.53 ppl=6.277202 [2023-12-17:12:49:12:INFO:sockeye.training:__call__] E=30 B=121500 s/sec=2123.25 tok/sec=99387.29 u/sec=23.24 ppl=6.282804 [2023-12-17:12:49:14:INFO:sockeye.training:__call__] E=30 B=121550 s/sec=2138.91 tok/sec=99746.93 u/sec=23.05 ppl=6.289426 [2023-12-17:12:49:17:INFO:sockeye.training:__call__] E=30 B=121600 s/sec=1882.64 tok/sec=101949.77 u/sec=23.03 ppl=6.293663 [2023-12-17:12:49:19:INFO:sockeye.training:__call__] E=30 B=121650 s/sec=2078.30 tok/sec=99849.19 u/sec=23.62 ppl=6.300416 [2023-12-17:12:49:21:INFO:sockeye.training:__call__] E=30 B=121700 s/sec=2089.61 tok/sec=99066.41 u/sec=23.53 ppl=6.307333 [2023-12-17:12:49:23:INFO:sockeye.training:__call__] E=30 B=121750 s/sec=2065.15 tok/sec=98069.86 u/sec=23.60 ppl=6.315204 [2023-12-17:12:49:25:INFO:sockeye.training:__call__] E=30 B=121800 s/sec=2236.25 tok/sec=96789.41 u/sec=23.89 ppl=6.323144 [2023-12-17:12:49:27:INFO:sockeye.training:__call__] E=30 B=121850 s/sec=2033.67 tok/sec=101307.27 u/sec=23.07 ppl=6.331324 [2023-12-17:12:49:29:INFO:sockeye.training:__call__] E=30 B=121900 s/sec=2156.06 tok/sec=97121.56 u/sec=23.81 ppl=6.337424 [2023-12-17:12:49:32:INFO:sockeye.training:__call__] E=30 B=121950 s/sec=2042.66 tok/sec=98910.88 u/sec=23.25 ppl=6.341862 [2023-12-17:12:49:34:INFO:sockeye.training:__call__] E=30 B=122000 s/sec=1920.51 tok/sec=99407.73 u/sec=23.13 ppl=6.346966 [2023-12-17:12:49:36:INFO:sockeye.training:__call__] E=30 B=122050 s/sec=2257.03 tok/sec=96671.73 u/sec=23.55 ppl=6.355797 [2023-12-17:12:49:38:INFO:sockeye.training:__call__] E=30 B=122100 s/sec=2308.88 tok/sec=98127.56 u/sec=23.73 ppl=6.361303 [2023-12-17:12:49:40:INFO:sockeye.training:__call__] E=30 B=122150 s/sec=2028.10 tok/sec=98042.40 u/sec=23.56 ppl=6.367209 [2023-12-17:12:49:42:INFO:sockeye.training:__call__] E=30 B=122200 s/sec=1846.58 tok/sec=101805.65 u/sec=22.76 ppl=6.371346 [2023-12-17:12:49:44:INFO:sockeye.training:__call__] E=30 B=122250 s/sec=1931.92 tok/sec=102119.99 u/sec=22.36 ppl=6.376792 [2023-12-17:12:49:47:INFO:sockeye.training:__call__] E=30 B=122300 s/sec=2105.94 tok/sec=99357.76 u/sec=23.67 ppl=6.381665 [2023-12-17:12:49:49:INFO:sockeye.training:__call__] E=30 B=122350 s/sec=2110.43 tok/sec=99321.44 u/sec=23.43 ppl=6.387562 [2023-12-17:12:49:51:INFO:sockeye.training:__call__] E=30 B=122400 s/sec=2118.86 tok/sec=101588.21 u/sec=22.91 ppl=6.394936 [2023-12-17:12:49:53:INFO:sockeye.training:__call__] E=30 B=122450 s/sec=2017.92 tok/sec=102487.97 u/sec=23.14 ppl=6.399089 [2023-12-17:12:49:55:INFO:sockeye.training:__call__] E=30 B=122500 s/sec=2213.74 tok/sec=95153.00 u/sec=23.77 ppl=6.406726 [2023-12-17:12:49:57:INFO:sockeye.training:__call__] E=30 B=122550 s/sec=2282.94 tok/sec=92905.88 u/sec=23.86 ppl=6.415493 [2023-12-17:12:49:59:INFO:sockeye.training:__call__] E=30 B=122600 s/sec=1903.55 tok/sec=101874.71 u/sec=22.92 ppl=6.419243 [2023-12-17:12:50:02:INFO:sockeye.training:__call__] E=30 B=122650 s/sec=2038.65 tok/sec=100262.25 u/sec=23.51 ppl=6.425460 [2023-12-17:12:50:04:INFO:sockeye.training:__call__] E=30 B=122700 s/sec=1942.36 tok/sec=103572.99 u/sec=22.69 ppl=6.429802 [2023-12-17:12:50:06:INFO:sockeye.training:__call__] E=30 B=122750 s/sec=1825.53 tok/sec=101729.37 u/sec=23.00 ppl=6.433258 [2023-12-17:12:50:08:INFO:sockeye.training:__call__] E=30 B=122800 s/sec=2137.76 tok/sec=101098.11 u/sec=23.20 ppl=6.440060 [2023-12-17:12:50:10:INFO:sockeye.training:__call__] E=30 B=122850 s/sec=1940.87 tok/sec=101146.32 u/sec=23.02 ppl=6.444306 [2023-12-17:12:50:12:INFO:sockeye.training:__call__] E=30 B=122900 s/sec=2059.23 tok/sec=99949.25 u/sec=23.36 ppl=6.448955 [2023-12-17:12:50:15:INFO:sockeye.training:__call__] E=30 B=122950 s/sec=1926.93 tok/sec=103404.92 u/sec=22.55 ppl=6.454321 [2023-12-17:12:50:17:INFO:sockeye.training:__call__] E=30 B=123000 s/sec=2230.15 tok/sec=96710.30 u/sec=23.54 ppl=6.461241 [2023-12-17:12:50:19:INFO:sockeye.training:__call__] E=30 B=123050 s/sec=1923.20 tok/sec=101568.22 u/sec=22.81 ppl=6.465424 [2023-12-17:12:50:21:INFO:sockeye.training:__call__] E=30 B=123100 s/sec=2081.21 tok/sec=99180.79 u/sec=23.35 ppl=6.471900 [2023-12-17:12:50:23:INFO:sockeye.training:__call__] E=30 B=123150 s/sec=2233.36 tok/sec=98327.44 u/sec=23.26 ppl=6.477397 [2023-12-17:12:50:25:INFO:sockeye.training:__call__] E=30 B=123200 s/sec=1938.39 tok/sec=100691.92 u/sec=23.34 ppl=6.481510 [2023-12-17:12:50:28:INFO:sockeye.training:__call__] E=30 B=123250 s/sec=2063.22 tok/sec=99758.62 u/sec=23.57 ppl=6.486472 [2023-12-17:12:50:30:INFO:sockeye.training:__call__] E=30 B=123300 s/sec=1927.26 tok/sec=102235.66 u/sec=22.68 ppl=6.491170 [2023-12-17:12:50:32:INFO:sockeye.training:__call__] E=30 B=123350 s/sec=1987.18 tok/sec=99433.46 u/sec=23.26 ppl=6.496745 [2023-12-17:12:50:34:INFO:sockeye.training:__call__] E=30 B=123400 s/sec=2310.41 tok/sec=97120.69 u/sec=23.71 ppl=6.503223 [2023-12-17:12:50:36:INFO:sockeye.training:__call__] E=30 B=123450 s/sec=2034.02 tok/sec=98147.37 u/sec=23.63 ppl=6.507832 [2023-12-17:12:50:38:INFO:sockeye.training:__call__] E=30 B=123500 s/sec=2076.34 tok/sec=101441.72 u/sec=22.77 ppl=6.514132 [2023-12-17:12:50:40:INFO:sockeye.training:__call__] E=30 B=123550 s/sec=2183.46 tok/sec=99837.03 u/sec=23.53 ppl=6.520098 [2023-12-17:12:50:43:INFO:sockeye.training:__call__] E=30 B=123600 s/sec=2117.56 tok/sec=99170.50 u/sec=23.85 ppl=6.527101 [2023-12-17:12:50:45:INFO:sockeye.training:__call__] E=30 B=123650 s/sec=2022.09 tok/sec=97029.95 u/sec=23.62 ppl=6.531397 [2023-12-17:12:50:47:INFO:sockeye.training:__call__] E=30 B=123700 s/sec=2067.28 tok/sec=96083.42 u/sec=23.41 ppl=6.535777 [2023-12-17:12:50:49:INFO:sockeye.training:__call__] E=30 B=123750 s/sec=2036.81 tok/sec=99659.38 u/sec=23.10 ppl=6.540494 [2023-12-17:12:50:51:INFO:sockeye.training:__call__] E=30 B=123800 s/sec=2016.21 tok/sec=102096.38 u/sec=22.71 ppl=6.544403 [2023-12-17:12:50:53:INFO:sockeye.training:__call__] E=31 B=123850 s/sec=2167.13 tok/sec=90187.46 u/sec=22.61 ppl=6.548731 [2023-12-17:12:50:56:INFO:sockeye.training:__call__] E=31 B=123900 s/sec=1968.46 tok/sec=102155.87 u/sec=22.78 ppl=6.538768 [2023-12-17:12:50:58:INFO:sockeye.training:__call__] E=31 B=123950 s/sec=1837.34 tok/sec=104717.90 u/sec=22.30 ppl=6.528820 [2023-12-17:12:51:00:INFO:sockeye.training:__call__] E=31 B=124000 s/sec=1853.70 tok/sec=103430.80 u/sec=22.63 ppl=6.520493 [2023-12-17:12:51:00:INFO:sockeye.training:_create_checkpoint] Checkpoint [31] Updates=124000 Epoch=31 Samples=11050200 Time-cost=172.361 Updates/sec=23.207 [2023-12-17:12:51:00:INFO:sockeye.training:_create_checkpoint] Checkpoint [31] Train-ppl=6.520493 [2023-12-17:12:51:03:INFO:sockeye.training:_evaluate] Checkpoint [31] Validation-ppl=32.513231 Validation-bleu=0.162242 Validation-chrf=30.272154 Validation-rouge1=0.158243 Validation-rouge2=0.074517 Validation-rougel=0.151255 Validation-length-ratio-mse=1.291628 Validation-ter=97.276104 Validation-avg-sec-per-sent=0.003340 Validation-decode-walltime=1.670135 [2023-12-17:12:51:03:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 7 checkpoints, best so far: 32.780032 [2023-12-17:12:51:03:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:12:51:03:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [31] Learning-rate=0.000162 [2023-12-17:12:51:03:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:51:03:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=7, base_lr=0.0002, lr=0.000162, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:51:03:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00031" [2023-12-17:12:51:06:INFO:sockeye.training:__call__] E=31 B=124050 s/sec=800.86 tok/sec=38992.41 u/sec=8.97 ppl=5.939929 [2023-12-17:12:51:08:INFO:sockeye.training:__call__] E=31 B=124100 s/sec=2090.04 tok/sec=95514.17 u/sec=23.92 ppl=5.942169 [2023-12-17:12:51:10:INFO:sockeye.training:__call__] E=31 B=124150 s/sec=2154.69 tok/sec=98487.62 u/sec=23.10 ppl=5.949938 [2023-12-17:12:51:12:INFO:sockeye.training:__call__] E=31 B=124200 s/sec=2094.29 tok/sec=101231.51 u/sec=22.96 ppl=5.965230 [2023-12-17:12:51:14:INFO:sockeye.training:__call__] E=31 B=124250 s/sec=1954.65 tok/sec=101583.25 u/sec=22.96 ppl=5.975937 [2023-12-17:12:51:16:INFO:sockeye.training:__call__] E=31 B=124300 s/sec=2250.57 tok/sec=97496.98 u/sec=23.64 ppl=5.978624 [2023-12-17:12:51:18:INFO:sockeye.training:__call__] E=31 B=124350 s/sec=2133.78 tok/sec=97776.73 u/sec=23.56 ppl=5.989525 [2023-12-17:12:51:21:INFO:sockeye.training:__call__] E=31 B=124400 s/sec=2097.76 tok/sec=97815.33 u/sec=23.41 ppl=6.000140 [2023-12-17:12:51:23:INFO:sockeye.training:__call__] E=31 B=124450 s/sec=2018.43 tok/sec=100387.42 u/sec=23.19 ppl=6.003626 [2023-12-17:12:51:25:INFO:sockeye.training:__call__] E=31 B=124500 s/sec=2099.65 tok/sec=96675.81 u/sec=23.95 ppl=6.005359 [2023-12-17:12:51:27:INFO:sockeye.training:__call__] E=31 B=124550 s/sec=2373.10 tok/sec=95905.38 u/sec=23.96 ppl=6.023998 [2023-12-17:12:51:29:INFO:sockeye.training:__call__] E=31 B=124600 s/sec=2091.27 tok/sec=97760.29 u/sec=23.81 ppl=6.039225 [2023-12-17:12:51:31:INFO:sockeye.training:__call__] E=31 B=124650 s/sec=2035.72 tok/sec=108711.72 u/sec=21.86 ppl=6.046731 [2023-12-17:12:51:33:INFO:sockeye.training:__call__] E=31 B=124700 s/sec=2165.73 tok/sec=96820.97 u/sec=23.42 ppl=6.056066 [2023-12-17:12:51:36:INFO:sockeye.training:__call__] E=31 B=124750 s/sec=2043.61 tok/sec=98003.68 u/sec=23.31 ppl=6.065608 [2023-12-17:12:51:38:INFO:sockeye.training:__call__] E=31 B=124800 s/sec=2078.69 tok/sec=98226.78 u/sec=23.75 ppl=6.073669 [2023-12-17:12:51:40:INFO:sockeye.training:__call__] E=31 B=124850 s/sec=2036.99 tok/sec=100154.65 u/sec=23.23 ppl=6.078134 [2023-12-17:12:51:42:INFO:sockeye.training:__call__] E=31 B=124900 s/sec=2228.59 tok/sec=98582.15 u/sec=23.41 ppl=6.089361 [2023-12-17:12:51:44:INFO:sockeye.training:__call__] E=31 B=124950 s/sec=2227.85 tok/sec=96843.36 u/sec=23.68 ppl=6.100649 [2023-12-17:12:51:46:INFO:sockeye.training:__call__] E=31 B=125000 s/sec=2099.62 tok/sec=98462.69 u/sec=23.23 ppl=6.108906 [2023-12-17:12:51:48:INFO:sockeye.training:__call__] E=31 B=125050 s/sec=2082.55 tok/sec=98603.49 u/sec=23.08 ppl=6.112743 [2023-12-17:12:51:51:INFO:sockeye.training:__call__] E=31 B=125100 s/sec=2209.86 tok/sec=96765.53 u/sec=23.61 ppl=6.124427 [2023-12-17:12:51:53:INFO:sockeye.training:__call__] E=31 B=125150 s/sec=2089.63 tok/sec=97435.64 u/sec=23.57 ppl=6.129604 [2023-12-17:12:51:55:INFO:sockeye.training:__call__] E=31 B=125200 s/sec=2146.14 tok/sec=98871.67 u/sec=23.29 ppl=6.135539 [2023-12-17:12:51:57:INFO:sockeye.training:__call__] E=31 B=125250 s/sec=2070.61 tok/sec=102022.05 u/sec=23.11 ppl=6.142744 [2023-12-17:12:51:59:INFO:sockeye.training:__call__] E=31 B=125300 s/sec=2126.49 tok/sec=99156.75 u/sec=23.24 ppl=6.151482 [2023-12-17:12:52:01:INFO:sockeye.training:__call__] E=31 B=125350 s/sec=2113.17 tok/sec=95666.13 u/sec=23.58 ppl=6.159249 [2023-12-17:12:52:03:INFO:sockeye.training:__call__] E=31 B=125400 s/sec=1923.43 tok/sec=104433.23 u/sec=22.47 ppl=6.165738 [2023-12-17:12:52:06:INFO:sockeye.training:__call__] E=31 B=125450 s/sec=1936.68 tok/sec=101427.66 u/sec=22.97 ppl=6.170181 [2023-12-17:12:52:08:INFO:sockeye.training:__call__] E=31 B=125500 s/sec=1972.76 tok/sec=99852.73 u/sec=23.58 ppl=6.177052 [2023-12-17:12:52:10:INFO:sockeye.training:__call__] E=31 B=125550 s/sec=2138.28 tok/sec=99263.91 u/sec=23.32 ppl=6.184903 [2023-12-17:12:52:12:INFO:sockeye.training:__call__] E=31 B=125600 s/sec=2017.88 tok/sec=99445.69 u/sec=22.97 ppl=6.192311 [2023-12-17:12:52:14:INFO:sockeye.training:__call__] E=31 B=125650 s/sec=2032.89 tok/sec=96810.10 u/sec=24.02 ppl=6.197351 [2023-12-17:12:52:16:INFO:sockeye.training:__call__] E=31 B=125700 s/sec=1838.70 tok/sec=102150.12 u/sec=22.80 ppl=6.202992 [2023-12-17:12:52:19:INFO:sockeye.training:__call__] E=31 B=125750 s/sec=1808.38 tok/sec=104697.24 u/sec=22.70 ppl=6.205059 [2023-12-17:12:52:21:INFO:sockeye.training:__call__] E=31 B=125800 s/sec=2259.75 tok/sec=98240.94 u/sec=23.50 ppl=6.213555 [2023-12-17:12:52:23:INFO:sockeye.training:__call__] E=31 B=125850 s/sec=2243.99 tok/sec=95236.65 u/sec=23.77 ppl=6.220610 [2023-12-17:12:52:25:INFO:sockeye.training:__call__] E=31 B=125900 s/sec=1973.23 tok/sec=100552.67 u/sec=23.27 ppl=6.225278 [2023-12-17:12:52:27:INFO:sockeye.training:__call__] E=31 B=125950 s/sec=2089.76 tok/sec=98832.69 u/sec=23.53 ppl=6.233761 [2023-12-17:12:52:29:INFO:sockeye.training:__call__] E=31 B=126000 s/sec=2250.00 tok/sec=98556.11 u/sec=23.13 ppl=6.241876 [2023-12-17:12:52:31:INFO:sockeye.training:__call__] E=31 B=126050 s/sec=2205.31 tok/sec=95132.23 u/sec=24.35 ppl=6.247478 [2023-12-17:12:52:33:INFO:sockeye.training:__call__] E=31 B=126100 s/sec=2157.39 tok/sec=98861.06 u/sec=23.21 ppl=6.253400 [2023-12-17:12:52:36:INFO:sockeye.training:__call__] E=31 B=126150 s/sec=2009.19 tok/sec=102805.63 u/sec=23.13 ppl=6.257214 [2023-12-17:12:52:38:INFO:sockeye.training:__call__] E=31 B=126200 s/sec=2225.62 tok/sec=99398.84 u/sec=23.22 ppl=6.263105 [2023-12-17:12:52:40:INFO:sockeye.training:__call__] E=31 B=126250 s/sec=2010.06 tok/sec=101408.11 u/sec=23.18 ppl=6.269051 [2023-12-17:12:52:42:INFO:sockeye.training:__call__] E=31 B=126300 s/sec=2021.89 tok/sec=100029.71 u/sec=23.27 ppl=6.274661 [2023-12-17:12:52:44:INFO:sockeye.training:__call__] E=31 B=126350 s/sec=1992.87 tok/sec=99672.56 u/sec=22.73 ppl=6.280313 [2023-12-17:12:52:46:INFO:sockeye.training:__call__] E=31 B=126400 s/sec=2081.29 tok/sec=100170.39 u/sec=23.31 ppl=6.286879 [2023-12-17:12:52:49:INFO:sockeye.training:__call__] E=31 B=126450 s/sec=2127.15 tok/sec=97344.37 u/sec=23.53 ppl=6.293924 [2023-12-17:12:52:51:INFO:sockeye.training:__call__] E=31 B=126500 s/sec=2165.99 tok/sec=100172.43 u/sec=22.98 ppl=6.299711 [2023-12-17:12:52:53:INFO:sockeye.training:__call__] E=31 B=126550 s/sec=2053.87 tok/sec=99485.03 u/sec=23.42 ppl=6.304812 [2023-12-17:12:52:55:INFO:sockeye.training:__call__] E=31 B=126600 s/sec=2108.23 tok/sec=100840.00 u/sec=23.12 ppl=6.310980 [2023-12-17:12:52:57:INFO:sockeye.training:__call__] E=31 B=126650 s/sec=2047.48 tok/sec=97269.59 u/sec=22.53 ppl=6.315139 [2023-12-17:12:52:59:INFO:sockeye.training:__call__] E=31 B=126700 s/sec=1986.88 tok/sec=98061.81 u/sec=22.14 ppl=6.319921 [2023-12-17:12:53:02:INFO:sockeye.training:__call__] E=31 B=126750 s/sec=1881.83 tok/sec=101122.22 u/sec=22.49 ppl=6.324184 [2023-12-17:12:53:04:INFO:sockeye.training:__call__] E=31 B=126800 s/sec=1908.95 tok/sec=102296.45 u/sec=22.77 ppl=6.327793 [2023-12-17:12:53:06:INFO:sockeye.training:__call__] E=31 B=126850 s/sec=2176.96 tok/sec=95923.69 u/sec=23.83 ppl=6.334134 [2023-12-17:12:53:08:INFO:sockeye.training:__call__] E=31 B=126900 s/sec=2040.02 tok/sec=100789.02 u/sec=23.10 ppl=6.339126 [2023-12-17:12:53:10:INFO:sockeye.training:__call__] E=31 B=126950 s/sec=2024.17 tok/sec=100920.05 u/sec=23.13 ppl=6.344355 [2023-12-17:12:53:12:INFO:sockeye.training:__call__] E=31 B=127000 s/sec=2008.30 tok/sec=101212.44 u/sec=23.07 ppl=6.349428 [2023-12-17:12:53:15:INFO:sockeye.training:__call__] E=31 B=127050 s/sec=2010.19 tok/sec=101523.70 u/sec=22.48 ppl=6.356045 [2023-12-17:12:53:17:INFO:sockeye.training:__call__] E=31 B=127100 s/sec=2106.30 tok/sec=94530.25 u/sec=22.62 ppl=6.360590 [2023-12-17:12:53:19:INFO:sockeye.training:__call__] E=31 B=127150 s/sec=2171.14 tok/sec=92898.68 u/sec=22.84 ppl=6.366728 [2023-12-17:12:53:21:INFO:sockeye.training:__call__] E=31 B=127200 s/sec=2086.91 tok/sec=98417.59 u/sec=23.63 ppl=6.373367 [2023-12-17:12:53:23:INFO:sockeye.training:__call__] E=31 B=127250 s/sec=2000.90 tok/sec=98055.09 u/sec=22.45 ppl=6.378579 [2023-12-17:12:53:26:INFO:sockeye.training:__call__] E=31 B=127300 s/sec=2113.63 tok/sec=95205.71 u/sec=23.14 ppl=6.385454 [2023-12-17:12:53:28:INFO:sockeye.training:__call__] E=31 B=127350 s/sec=1743.23 tok/sec=104957.84 u/sec=21.88 ppl=6.388661 [2023-12-17:12:53:30:INFO:sockeye.training:__call__] E=31 B=127400 s/sec=2175.91 tok/sec=95620.03 u/sec=23.45 ppl=6.394994 [2023-12-17:12:53:32:INFO:sockeye.training:__call__] E=31 B=127450 s/sec=2122.35 tok/sec=95165.24 u/sec=23.39 ppl=6.400450 [2023-12-17:12:53:34:INFO:sockeye.training:__call__] E=31 B=127500 s/sec=1987.79 tok/sec=99745.32 u/sec=23.18 ppl=6.405083 [2023-12-17:12:53:36:INFO:sockeye.training:__call__] E=31 B=127550 s/sec=2307.85 tok/sec=95257.08 u/sec=24.20 ppl=6.411771 [2023-12-17:12:53:38:INFO:sockeye.training:__call__] E=31 B=127600 s/sec=2202.18 tok/sec=96451.55 u/sec=23.94 ppl=6.417297 [2023-12-17:12:53:41:INFO:sockeye.training:__call__] E=31 B=127650 s/sec=1941.24 tok/sec=100328.87 u/sec=23.47 ppl=6.420641 [2023-12-17:12:53:43:INFO:sockeye.training:__call__] E=31 B=127700 s/sec=1986.90 tok/sec=97304.60 u/sec=22.21 ppl=6.423993 [2023-12-17:12:53:45:INFO:sockeye.training:__call__] E=31 B=127750 s/sec=1846.01 tok/sec=99362.14 u/sec=22.40 ppl=6.428173 [2023-12-17:12:53:47:INFO:sockeye.training:__call__] E=31 B=127800 s/sec=2010.97 tok/sec=97145.39 u/sec=23.02 ppl=6.434289 [2023-12-17:12:53:50:INFO:sockeye.training:__call__] E=32 B=127850 s/sec=1858.11 tok/sec=89298.35 u/sec=21.27 ppl=6.437334 [2023-12-17:12:53:52:INFO:sockeye.training:__call__] E=32 B=127900 s/sec=2122.66 tok/sec=96522.04 u/sec=23.82 ppl=6.429154 [2023-12-17:12:53:54:INFO:sockeye.training:__call__] E=32 B=127950 s/sec=1730.47 tok/sec=104192.41 u/sec=22.77 ppl=6.419698 [2023-12-17:12:53:56:INFO:sockeye.training:__call__] E=32 B=128000 s/sec=2087.93 tok/sec=99322.47 u/sec=23.39 ppl=6.412820 [2023-12-17:12:53:56:INFO:sockeye.training:_create_checkpoint] Checkpoint [32] Updates=128000 Epoch=32 Samples=11406968 Time-cost=172.644 Updates/sec=23.169 [2023-12-17:12:53:56:INFO:sockeye.training:_create_checkpoint] Checkpoint [32] Train-ppl=6.412820 [2023-12-17:12:53:59:INFO:sockeye.training:_evaluate] Checkpoint [32] Validation-ppl=33.819907 Validation-bleu=0.194085 Validation-chrf=32.498671 Validation-rouge1=0.167656 Validation-rouge2=0.085043 Validation-rougel=0.159627 Validation-length-ratio-mse=1.309333 Validation-ter=95.996698 Validation-avg-sec-per-sent=0.003667 Validation-decode-walltime=1.833360 [2023-12-17:12:53:59:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 8 checkpoints, best so far: 32.780032 [2023-12-17:12:53:59:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:12:53:59:INFO:sockeye.lr_scheduler:new_evaluation_result] 8 checkpoints since improvement or rate scaling, lowering learning rate: 1.62e-04 -> 1.46e-04 [2023-12-17:12:53:59:INFO:sockeye.training:_adjust_learning_rate] Loading model parameters and optimizer states from best checkpoint: 24 [2023-12-17:12:53:59:INFO:sockeye.model:load_parameters] Loaded params from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best" to "cuda:0" [2023-12-17:12:53:59:INFO:sockeye.training:_load_optimizer_state] Loaded optimizer state from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl" [2023-12-17:12:53:59:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [32] Learning-rate=0.000162 [2023-12-17:12:54:00:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:54:00:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:54:00:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00032" [2023-12-17:12:54:02:INFO:sockeye.training:__call__] E=32 B=128050 s/sec=785.34 tok/sec=32926.20 u/sec=8.26 ppl=6.996054 [2023-12-17:12:54:04:INFO:sockeye.training:__call__] E=32 B=128100 s/sec=1914.14 tok/sec=102693.12 u/sec=22.53 ppl=6.887231 [2023-12-17:12:54:06:INFO:sockeye.training:__call__] E=32 B=128150 s/sec=2100.78 tok/sec=96979.56 u/sec=23.36 ppl=6.882953 [2023-12-17:12:54:09:INFO:sockeye.training:__call__] E=32 B=128200 s/sec=1808.31 tok/sec=100543.48 u/sec=23.50 ppl=6.883384 [2023-12-17:12:54:11:INFO:sockeye.training:__call__] E=32 B=128250 s/sec=1951.58 tok/sec=100176.38 u/sec=23.19 ppl=6.887496 [2023-12-17:12:54:13:INFO:sockeye.training:__call__] E=32 B=128300 s/sec=1987.26 tok/sec=100737.69 u/sec=23.35 ppl=6.891119 [2023-12-17:12:54:15:INFO:sockeye.training:__call__] E=32 B=128350 s/sec=2124.66 tok/sec=99525.43 u/sec=23.42 ppl=6.908195 [2023-12-17:12:54:17:INFO:sockeye.training:__call__] E=32 B=128400 s/sec=2066.99 tok/sec=101231.42 u/sec=22.82 ppl=6.920443 [2023-12-17:12:54:19:INFO:sockeye.training:__call__] E=32 B=128450 s/sec=1870.30 tok/sec=100487.74 u/sec=23.38 ppl=6.914404 [2023-12-17:12:54:22:INFO:sockeye.training:__call__] E=32 B=128500 s/sec=2042.53 tok/sec=100477.74 u/sec=23.21 ppl=6.922321 [2023-12-17:12:54:24:INFO:sockeye.training:__call__] E=32 B=128550 s/sec=2427.35 tok/sec=94354.94 u/sec=24.04 ppl=6.949002 [2023-12-17:12:54:26:INFO:sockeye.training:__call__] E=32 B=128600 s/sec=2206.64 tok/sec=99955.40 u/sec=23.06 ppl=6.962220 [2023-12-17:12:54:28:INFO:sockeye.training:__call__] E=32 B=128650 s/sec=2024.29 tok/sec=98176.36 u/sec=23.21 ppl=6.974773 [2023-12-17:12:54:30:INFO:sockeye.training:__call__] E=32 B=128700 s/sec=2112.15 tok/sec=100009.97 u/sec=23.32 ppl=6.984411 [2023-12-17:12:54:32:INFO:sockeye.training:__call__] E=32 B=128750 s/sec=1952.48 tok/sec=104450.46 u/sec=22.60 ppl=6.985552 [2023-12-17:12:54:34:INFO:sockeye.training:__call__] E=32 B=128800 s/sec=2264.15 tok/sec=96998.13 u/sec=23.70 ppl=7.003218 [2023-12-17:12:54:37:INFO:sockeye.training:__call__] E=32 B=128850 s/sec=2122.99 tok/sec=98799.32 u/sec=23.32 ppl=7.014239 [2023-12-17:12:54:39:INFO:sockeye.training:__call__] E=32 B=128900 s/sec=2093.62 tok/sec=95912.50 u/sec=23.24 ppl=7.024351 [2023-12-17:12:54:41:INFO:sockeye.training:__call__] E=32 B=128950 s/sec=2023.19 tok/sec=100870.76 u/sec=23.12 ppl=7.030060 [2023-12-17:12:54:43:INFO:sockeye.training:__call__] E=32 B=129000 s/sec=1950.87 tok/sec=101005.29 u/sec=22.92 ppl=7.033179 [2023-12-17:12:54:45:INFO:sockeye.training:__call__] E=32 B=129050 s/sec=1758.11 tok/sec=105472.07 u/sec=22.29 ppl=7.042317 [2023-12-17:12:54:47:INFO:sockeye.training:__call__] E=32 B=129100 s/sec=2085.70 tok/sec=100528.50 u/sec=23.15 ppl=7.051914 [2023-12-17:12:54:50:INFO:sockeye.training:__call__] E=32 B=129150 s/sec=2044.82 tok/sec=98332.17 u/sec=23.54 ppl=7.061000 [2023-12-17:12:54:52:INFO:sockeye.training:__call__] E=32 B=129200 s/sec=1939.40 tok/sec=100162.19 u/sec=22.83 ppl=7.068981 [2023-12-17:12:54:54:INFO:sockeye.training:__call__] E=32 B=129250 s/sec=2329.39 tok/sec=91620.08 u/sec=24.30 ppl=7.081442 [2023-12-17:12:54:56:INFO:sockeye.training:__call__] E=32 B=129300 s/sec=2240.53 tok/sec=99555.98 u/sec=23.38 ppl=7.090223 [2023-12-17:12:54:58:INFO:sockeye.training:__call__] E=32 B=129350 s/sec=2175.44 tok/sec=97838.50 u/sec=23.48 ppl=7.103118 [2023-12-17:12:55:00:INFO:sockeye.training:__call__] E=32 B=129400 s/sec=2339.22 tok/sec=94401.07 u/sec=24.09 ppl=7.120479 [2023-12-17:12:55:02:INFO:sockeye.training:__call__] E=32 B=129450 s/sec=2121.82 tok/sec=95966.91 u/sec=23.68 ppl=7.136084 [2023-12-17:12:55:04:INFO:sockeye.training:__call__] E=32 B=129500 s/sec=2150.13 tok/sec=96361.84 u/sec=23.45 ppl=7.145890 [2023-12-17:12:55:07:INFO:sockeye.training:__call__] E=32 B=129550 s/sec=2301.11 tok/sec=97040.74 u/sec=23.23 ppl=7.154214 [2023-12-17:12:55:09:INFO:sockeye.training:__call__] E=32 B=129600 s/sec=2107.57 tok/sec=103299.66 u/sec=22.56 ppl=7.163111 [2023-12-17:12:55:11:INFO:sockeye.training:__call__] E=32 B=129650 s/sec=2056.21 tok/sec=101651.34 u/sec=23.07 ppl=7.168136 [2023-12-17:12:55:13:INFO:sockeye.training:__call__] E=32 B=129700 s/sec=2130.68 tok/sec=100292.06 u/sec=23.44 ppl=7.175560 [2023-12-17:12:55:15:INFO:sockeye.training:__call__] E=32 B=129750 s/sec=1986.81 tok/sec=102304.25 u/sec=22.87 ppl=7.184171 [2023-12-17:12:55:17:INFO:sockeye.training:__call__] E=32 B=129800 s/sec=1919.34 tok/sec=99221.80 u/sec=23.38 ppl=7.190470 [2023-12-17:12:55:20:INFO:sockeye.training:__call__] E=32 B=129850 s/sec=2005.75 tok/sec=103052.45 u/sec=22.92 ppl=7.196196 [2023-12-17:12:55:22:INFO:sockeye.training:__call__] E=32 B=129900 s/sec=2192.25 tok/sec=97269.78 u/sec=23.46 ppl=7.204367 [2023-12-17:12:55:24:INFO:sockeye.training:__call__] E=32 B=129950 s/sec=1927.84 tok/sec=103729.37 u/sec=22.48 ppl=7.211709 [2023-12-17:12:55:26:INFO:sockeye.training:__call__] E=32 B=130000 s/sec=1790.78 tok/sec=101638.69 u/sec=22.52 ppl=7.219341 [2023-12-17:12:55:28:INFO:sockeye.training:__call__] E=32 B=130050 s/sec=2126.87 tok/sec=97880.76 u/sec=23.32 ppl=7.227915 [2023-12-17:12:55:30:INFO:sockeye.training:__call__] E=32 B=130100 s/sec=2033.69 tok/sec=101190.09 u/sec=23.07 ppl=7.235880 [2023-12-17:12:55:33:INFO:sockeye.training:__call__] E=32 B=130150 s/sec=2256.44 tok/sec=96938.77 u/sec=23.94 ppl=7.245216 [2023-12-17:12:55:35:INFO:sockeye.training:__call__] E=32 B=130200 s/sec=2029.53 tok/sec=99224.81 u/sec=23.53 ppl=7.252532 [2023-12-17:12:55:37:INFO:sockeye.training:__call__] E=32 B=130250 s/sec=2112.67 tok/sec=96552.97 u/sec=23.71 ppl=7.262039 [2023-12-17:12:55:39:INFO:sockeye.training:__call__] E=32 B=130300 s/sec=1919.14 tok/sec=101377.69 u/sec=22.89 ppl=7.265777 [2023-12-17:12:55:41:INFO:sockeye.training:__call__] E=32 B=130350 s/sec=2005.28 tok/sec=103576.90 u/sec=22.70 ppl=7.271218 [2023-12-17:12:55:43:INFO:sockeye.training:__call__] E=32 B=130400 s/sec=2235.00 tok/sec=95779.35 u/sec=23.96 ppl=7.280607 [2023-12-17:12:55:46:INFO:sockeye.training:__call__] E=32 B=130450 s/sec=1759.89 tok/sec=110673.01 u/sec=21.57 ppl=7.284384 [2023-12-17:12:55:48:INFO:sockeye.training:__call__] E=32 B=130500 s/sec=2094.98 tok/sec=99024.71 u/sec=23.51 ppl=7.290495 [2023-12-17:12:55:50:INFO:sockeye.training:__call__] E=32 B=130550 s/sec=1995.04 tok/sec=100309.40 u/sec=22.92 ppl=7.295493 [2023-12-17:12:55:52:INFO:sockeye.training:__call__] E=32 B=130600 s/sec=2193.99 tok/sec=96851.83 u/sec=23.52 ppl=7.303905 [2023-12-17:12:55:54:INFO:sockeye.training:__call__] E=32 B=130650 s/sec=2046.60 tok/sec=98028.09 u/sec=23.30 ppl=7.311559 [2023-12-17:12:55:56:INFO:sockeye.training:__call__] E=32 B=130700 s/sec=2150.68 tok/sec=98931.15 u/sec=23.18 ppl=7.319783 [2023-12-17:12:55:58:INFO:sockeye.training:__call__] E=32 B=130750 s/sec=1988.30 tok/sec=101421.88 u/sec=23.14 ppl=7.326070 [2023-12-17:12:56:01:INFO:sockeye.training:__call__] E=32 B=130800 s/sec=2170.21 tok/sec=97129.03 u/sec=23.51 ppl=7.335326 [2023-12-17:12:56:03:INFO:sockeye.training:__call__] E=32 B=130850 s/sec=2247.29 tok/sec=94161.85 u/sec=24.17 ppl=7.344741 [2023-12-17:12:56:05:INFO:sockeye.training:__call__] E=32 B=130900 s/sec=1960.61 tok/sec=100575.06 u/sec=23.39 ppl=7.350552 [2023-12-17:12:56:07:INFO:sockeye.training:__call__] E=32 B=130950 s/sec=1978.38 tok/sec=102300.44 u/sec=22.48 ppl=7.357275 [2023-12-17:12:56:09:INFO:sockeye.training:__call__] E=32 B=131000 s/sec=2317.09 tok/sec=97265.48 u/sec=23.40 ppl=7.366988 [2023-12-17:12:56:11:INFO:sockeye.training:__call__] E=32 B=131050 s/sec=2242.32 tok/sec=98558.31 u/sec=23.16 ppl=7.375553 [2023-12-17:12:56:13:INFO:sockeye.training:__call__] E=32 B=131100 s/sec=2115.97 tok/sec=95832.94 u/sec=23.70 ppl=7.381509 [2023-12-17:12:56:16:INFO:sockeye.training:__call__] E=32 B=131150 s/sec=1951.15 tok/sec=99430.68 u/sec=23.14 ppl=7.385404 [2023-12-17:12:56:18:INFO:sockeye.training:__call__] E=32 B=131200 s/sec=2056.39 tok/sec=99735.03 u/sec=22.95 ppl=7.392294 [2023-12-17:12:56:20:INFO:sockeye.training:__call__] E=32 B=131250 s/sec=1992.02 tok/sec=101124.10 u/sec=22.89 ppl=7.396039 [2023-12-17:12:56:22:INFO:sockeye.training:__call__] E=32 B=131300 s/sec=2151.55 tok/sec=97916.07 u/sec=23.39 ppl=7.404788 [2023-12-17:12:56:24:INFO:sockeye.training:__call__] E=32 B=131350 s/sec=1963.38 tok/sec=99674.78 u/sec=23.07 ppl=7.410381 [2023-12-17:12:56:26:INFO:sockeye.training:__call__] E=32 B=131400 s/sec=2128.89 tok/sec=97966.66 u/sec=23.72 ppl=7.414425 [2023-12-17:12:56:28:INFO:sockeye.training:__call__] E=32 B=131450 s/sec=2165.06 tok/sec=98835.90 u/sec=23.66 ppl=7.420313 [2023-12-17:12:56:31:INFO:sockeye.training:__call__] E=32 B=131500 s/sec=2075.47 tok/sec=100416.15 u/sec=22.96 ppl=7.426189 [2023-12-17:12:56:33:INFO:sockeye.training:__call__] E=32 B=131550 s/sec=2169.64 tok/sec=96058.82 u/sec=23.83 ppl=7.433141 [2023-12-17:12:56:35:INFO:sockeye.training:__call__] E=32 B=131600 s/sec=2080.60 tok/sec=100460.07 u/sec=23.10 ppl=7.440000 [2023-12-17:12:56:37:INFO:sockeye.training:__call__] E=32 B=131650 s/sec=2123.07 tok/sec=97402.53 u/sec=23.78 ppl=7.448191 [2023-12-17:12:56:39:INFO:sockeye.training:__call__] E=32 B=131700 s/sec=2029.90 tok/sec=102216.34 u/sec=22.78 ppl=7.453323 [2023-12-17:12:56:41:INFO:sockeye.training:__call__] E=32 B=131750 s/sec=1918.58 tok/sec=101998.69 u/sec=23.10 ppl=7.458733 [2023-12-17:12:56:44:INFO:sockeye.training:__call__] E=32 B=131800 s/sec=2128.86 tok/sec=98779.20 u/sec=23.14 ppl=7.465096 [2023-12-17:12:56:46:INFO:sockeye.training:__call__] E=33 B=131850 s/sec=2007.98 tok/sec=91761.34 u/sec=22.02 ppl=7.465968 [2023-12-17:12:56:48:INFO:sockeye.training:__call__] E=33 B=131900 s/sec=1986.20 tok/sec=99184.03 u/sec=23.12 ppl=7.454293 [2023-12-17:12:56:50:INFO:sockeye.training:__call__] E=33 B=131950 s/sec=1953.48 tok/sec=103266.85 u/sec=22.91 ppl=7.442066 [2023-12-17:12:56:52:INFO:sockeye.training:__call__] E=33 B=132000 s/sec=2141.52 tok/sec=97010.74 u/sec=23.90 ppl=7.431422 [2023-12-17:12:56:52:INFO:sockeye.training:_create_checkpoint] Checkpoint [33] Updates=132000 Epoch=33 Samples=11763832 Time-cost=172.248 Updates/sec=23.222 [2023-12-17:12:56:52:INFO:sockeye.training:_create_checkpoint] Checkpoint [33] Train-ppl=7.431422 [2023-12-17:12:56:55:INFO:sockeye.training:_evaluate] Checkpoint [33] Validation-ppl=29.339310 Validation-bleu=0.173656 Validation-chrf=31.774343 Validation-rouge1=0.169756 Validation-rouge2=0.079812 Validation-rougel=0.161685 Validation-length-ratio-mse=1.274281 Validation-ter=95.583987 Validation-avg-sec-per-sent=0.003300 Validation-decode-walltime=1.650136 [2023-12-17:12:56:55:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 9 checkpoints, best so far: 32.780032 [2023-12-17:12:56:55:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:12:56:55:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [33] Learning-rate=0.000146 [2023-12-17:12:56:55:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:56:55:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:56:56:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00033" [2023-12-17:12:56:58:INFO:sockeye.training:__call__] E=33 B=132050 s/sec=771.13 tok/sec=38042.63 u/sec=8.83 ppl=6.634690 [2023-12-17:12:57:00:INFO:sockeye.training:__call__] E=33 B=132100 s/sec=2128.83 tok/sec=101117.38 u/sec=23.14 ppl=6.737788 [2023-12-17:12:57:02:INFO:sockeye.training:__call__] E=33 B=132150 s/sec=2028.64 tok/sec=101088.35 u/sec=22.85 ppl=6.724889 [2023-12-17:12:57:04:INFO:sockeye.training:__call__] E=33 B=132200 s/sec=2148.41 tok/sec=99769.33 u/sec=23.19 ppl=6.741750 [2023-12-17:12:57:07:INFO:sockeye.training:__call__] E=33 B=132250 s/sec=2162.79 tok/sec=98857.63 u/sec=23.19 ppl=6.739020 [2023-12-17:12:57:09:INFO:sockeye.training:__call__] E=33 B=132300 s/sec=2056.35 tok/sec=100325.93 u/sec=23.45 ppl=6.763191 [2023-12-17:12:57:11:INFO:sockeye.training:__call__] E=33 B=132350 s/sec=1966.24 tok/sec=102883.17 u/sec=22.67 ppl=6.748824 [2023-12-17:12:57:13:INFO:sockeye.training:__call__] E=33 B=132400 s/sec=2132.40 tok/sec=102296.20 u/sec=23.10 ppl=6.763359 [2023-12-17:12:57:15:INFO:sockeye.training:__call__] E=33 B=132450 s/sec=1979.59 tok/sec=99120.66 u/sec=23.21 ppl=6.768259 [2023-12-17:12:57:17:INFO:sockeye.training:__call__] E=33 B=132500 s/sec=1992.51 tok/sec=100211.95 u/sec=23.19 ppl=6.774362 [2023-12-17:12:57:20:INFO:sockeye.training:__call__] E=33 B=132550 s/sec=2124.80 tok/sec=98041.97 u/sec=23.55 ppl=6.790502 [2023-12-17:12:57:22:INFO:sockeye.training:__call__] E=33 B=132600 s/sec=2352.09 tok/sec=95852.30 u/sec=23.83 ppl=6.811955 [2023-12-17:12:57:24:INFO:sockeye.training:__call__] E=33 B=132650 s/sec=2006.59 tok/sec=100500.29 u/sec=23.18 ppl=6.817039 [2023-12-17:12:57:26:INFO:sockeye.training:__call__] E=33 B=132700 s/sec=2260.19 tok/sec=96363.33 u/sec=23.98 ppl=6.828204 [2023-12-17:12:57:28:INFO:sockeye.training:__call__] E=33 B=132750 s/sec=2358.83 tok/sec=94898.30 u/sec=23.66 ppl=6.843222 [2023-12-17:12:57:30:INFO:sockeye.training:__call__] E=33 B=132800 s/sec=2073.78 tok/sec=96766.03 u/sec=24.09 ppl=6.850244 [2023-12-17:12:57:32:INFO:sockeye.training:__call__] E=33 B=132850 s/sec=2179.30 tok/sec=97674.89 u/sec=23.65 ppl=6.858884 [2023-12-17:12:57:34:INFO:sockeye.training:__call__] E=33 B=132900 s/sec=2111.96 tok/sec=100814.92 u/sec=23.00 ppl=6.865022 [2023-12-17:12:57:36:INFO:sockeye.training:__call__] E=33 B=132950 s/sec=2308.50 tok/sec=93732.01 u/sec=24.17 ppl=6.880340 [2023-12-17:12:57:39:INFO:sockeye.training:__call__] E=33 B=133000 s/sec=1801.08 tok/sec=105376.04 u/sec=22.47 ppl=6.885011 [2023-12-17:12:57:41:INFO:sockeye.training:__call__] E=33 B=133050 s/sec=1923.78 tok/sec=102212.36 u/sec=22.82 ppl=6.893814 [2023-12-17:12:57:43:INFO:sockeye.training:__call__] E=33 B=133100 s/sec=1978.80 tok/sec=102513.76 u/sec=23.07 ppl=6.898028 [2023-12-17:12:57:45:INFO:sockeye.training:__call__] E=33 B=133150 s/sec=2308.87 tok/sec=93098.61 u/sec=24.21 ppl=6.909007 [2023-12-17:12:57:47:INFO:sockeye.training:__call__] E=33 B=133200 s/sec=2159.97 tok/sec=98603.54 u/sec=23.60 ppl=6.917596 [2023-12-17:12:57:49:INFO:sockeye.training:__call__] E=33 B=133250 s/sec=1993.57 tok/sec=106897.01 u/sec=21.67 ppl=6.926915 [2023-12-17:12:57:52:INFO:sockeye.training:__call__] E=33 B=133300 s/sec=1801.89 tok/sec=100701.65 u/sec=22.80 ppl=6.932545 [2023-12-17:12:57:54:INFO:sockeye.training:__call__] E=33 B=133350 s/sec=2214.20 tok/sec=97000.05 u/sec=23.70 ppl=6.940896 [2023-12-17:12:57:56:INFO:sockeye.training:__call__] E=33 B=133400 s/sec=2098.00 tok/sec=99911.70 u/sec=22.92 ppl=6.946162 [2023-12-17:12:57:58:INFO:sockeye.training:__call__] E=33 B=133450 s/sec=2079.77 tok/sec=98472.52 u/sec=23.55 ppl=6.954082 [2023-12-17:12:58:00:INFO:sockeye.training:__call__] E=33 B=133500 s/sec=2241.99 tok/sec=98301.10 u/sec=23.55 ppl=6.962445 [2023-12-17:12:58:02:INFO:sockeye.training:__call__] E=33 B=133550 s/sec=2177.17 tok/sec=97126.13 u/sec=23.62 ppl=6.967945 [2023-12-17:12:58:04:INFO:sockeye.training:__call__] E=33 B=133600 s/sec=2060.15 tok/sec=99304.33 u/sec=23.28 ppl=6.972753 [2023-12-17:12:58:07:INFO:sockeye.training:__call__] E=33 B=133650 s/sec=2463.03 tok/sec=96389.46 u/sec=23.79 ppl=6.987339 [2023-12-17:12:58:09:INFO:sockeye.training:__call__] E=33 B=133700 s/sec=1815.07 tok/sec=104218.24 u/sec=22.60 ppl=6.990759 [2023-12-17:12:58:11:INFO:sockeye.training:__call__] E=33 B=133750 s/sec=2112.83 tok/sec=99417.13 u/sec=22.97 ppl=7.000052 [2023-12-17:12:58:13:INFO:sockeye.training:__call__] E=33 B=133800 s/sec=1891.42 tok/sec=103766.11 u/sec=22.43 ppl=7.004530 [2023-12-17:12:58:15:INFO:sockeye.training:__call__] E=33 B=133850 s/sec=2091.77 tok/sec=99986.48 u/sec=23.35 ppl=7.012973 [2023-12-17:12:58:18:INFO:sockeye.training:__call__] E=33 B=133900 s/sec=1973.64 tok/sec=102019.82 u/sec=22.68 ppl=7.018499 [2023-12-17:12:58:20:INFO:sockeye.training:__call__] E=33 B=133950 s/sec=1932.99 tok/sec=100937.95 u/sec=22.75 ppl=7.021950 [2023-12-17:12:58:22:INFO:sockeye.training:__call__] E=33 B=134000 s/sec=2235.17 tok/sec=97826.97 u/sec=23.24 ppl=7.030823 [2023-12-17:12:58:24:INFO:sockeye.training:__call__] E=33 B=134050 s/sec=1913.44 tok/sec=99904.77 u/sec=23.50 ppl=7.035194 [2023-12-17:12:58:26:INFO:sockeye.training:__call__] E=33 B=134100 s/sec=2142.30 tok/sec=95617.62 u/sec=24.08 ppl=7.039610 [2023-12-17:12:58:28:INFO:sockeye.training:__call__] E=33 B=134150 s/sec=2039.17 tok/sec=98815.09 u/sec=23.56 ppl=7.045351 [2023-12-17:12:58:30:INFO:sockeye.training:__call__] E=33 B=134200 s/sec=1997.89 tok/sec=98638.07 u/sec=23.52 ppl=7.050049 [2023-12-17:12:58:33:INFO:sockeye.training:__call__] E=33 B=134250 s/sec=1898.05 tok/sec=102363.04 u/sec=22.90 ppl=7.055808 [2023-12-17:12:58:35:INFO:sockeye.training:__call__] E=33 B=134300 s/sec=2278.32 tok/sec=96564.44 u/sec=23.58 ppl=7.064098 [2023-12-17:12:58:37:INFO:sockeye.training:__call__] E=33 B=134350 s/sec=2000.95 tok/sec=99973.14 u/sec=23.25 ppl=7.070648 [2023-12-17:12:58:39:INFO:sockeye.training:__call__] E=33 B=134400 s/sec=1725.29 tok/sec=105204.42 u/sec=21.78 ppl=7.075455 [2023-12-17:12:58:41:INFO:sockeye.training:__call__] E=33 B=134450 s/sec=2106.33 tok/sec=97854.02 u/sec=23.51 ppl=7.083734 [2023-12-17:12:58:43:INFO:sockeye.training:__call__] E=33 B=134500 s/sec=1898.64 tok/sec=100085.43 u/sec=23.54 ppl=7.087829 [2023-12-17:12:58:46:INFO:sockeye.training:__call__] E=33 B=134550 s/sec=1855.18 tok/sec=99649.53 u/sec=23.33 ppl=7.092572 [2023-12-17:12:58:48:INFO:sockeye.training:__call__] E=33 B=134600 s/sec=2062.26 tok/sec=101855.00 u/sec=22.85 ppl=7.099014 [2023-12-17:12:58:50:INFO:sockeye.training:__call__] E=33 B=134650 s/sec=2009.05 tok/sec=98670.83 u/sec=23.30 ppl=7.104384 [2023-12-17:12:58:52:INFO:sockeye.training:__call__] E=33 B=134700 s/sec=2034.17 tok/sec=100518.70 u/sec=22.95 ppl=7.111581 [2023-12-17:12:58:54:INFO:sockeye.training:__call__] E=33 B=134750 s/sec=1921.43 tok/sec=100741.90 u/sec=23.09 ppl=7.118197 [2023-12-17:12:58:56:INFO:sockeye.training:__call__] E=33 B=134800 s/sec=1866.64 tok/sec=103111.26 u/sec=22.65 ppl=7.121666 [2023-12-17:12:58:59:INFO:sockeye.training:__call__] E=33 B=134850 s/sec=2005.88 tok/sec=101016.01 u/sec=23.26 ppl=7.130681 [2023-12-17:12:59:01:INFO:sockeye.training:__call__] E=33 B=134900 s/sec=2129.93 tok/sec=98426.62 u/sec=23.44 ppl=7.138944 [2023-12-17:12:59:03:INFO:sockeye.training:__call__] E=33 B=134950 s/sec=2229.61 tok/sec=95508.67 u/sec=23.98 ppl=7.146011 [2023-12-17:12:59:05:INFO:sockeye.training:__call__] E=33 B=135000 s/sec=2451.47 tok/sec=93368.84 u/sec=23.79 ppl=7.155591 [2023-12-17:12:59:07:INFO:sockeye.training:__call__] E=33 B=135050 s/sec=2033.47 tok/sec=98382.42 u/sec=23.32 ppl=7.161007 [2023-12-17:12:59:09:INFO:sockeye.training:__call__] E=33 B=135100 s/sec=2108.99 tok/sec=101499.66 u/sec=23.29 ppl=7.166676 [2023-12-17:12:59:11:INFO:sockeye.training:__call__] E=33 B=135150 s/sec=1918.51 tok/sec=100837.69 u/sec=23.01 ppl=7.169295 [2023-12-17:12:59:13:INFO:sockeye.training:__call__] E=33 B=135200 s/sec=2111.18 tok/sec=100236.29 u/sec=23.23 ppl=7.174106 [2023-12-17:12:59:16:INFO:sockeye.training:__call__] E=33 B=135250 s/sec=2118.64 tok/sec=99851.06 u/sec=23.23 ppl=7.180548 [2023-12-17:12:59:18:INFO:sockeye.training:__call__] E=33 B=135300 s/sec=2128.70 tok/sec=98896.02 u/sec=23.10 ppl=7.185573 [2023-12-17:12:59:20:INFO:sockeye.training:__call__] E=33 B=135350 s/sec=2217.60 tok/sec=97723.09 u/sec=23.26 ppl=7.191733 [2023-12-17:12:59:22:INFO:sockeye.training:__call__] E=33 B=135400 s/sec=2152.13 tok/sec=100032.83 u/sec=23.43 ppl=7.196488 [2023-12-17:12:59:24:INFO:sockeye.training:__call__] E=33 B=135450 s/sec=2273.89 tok/sec=96576.03 u/sec=23.61 ppl=7.204728 [2023-12-17:12:59:26:INFO:sockeye.training:__call__] E=33 B=135500 s/sec=2137.60 tok/sec=97804.77 u/sec=23.44 ppl=7.210339 [2023-12-17:12:59:28:INFO:sockeye.training:__call__] E=33 B=135550 s/sec=1968.16 tok/sec=98727.27 u/sec=23.75 ppl=7.214164 [2023-12-17:12:59:31:INFO:sockeye.training:__call__] E=33 B=135600 s/sec=2293.05 tok/sec=97786.72 u/sec=23.46 ppl=7.220515 [2023-12-17:12:59:33:INFO:sockeye.training:__call__] E=33 B=135650 s/sec=2154.30 tok/sec=96368.30 u/sec=23.50 ppl=7.227462 [2023-12-17:12:59:35:INFO:sockeye.training:__call__] E=33 B=135700 s/sec=1770.91 tok/sec=102367.93 u/sec=22.92 ppl=7.231736 [2023-12-17:12:59:37:INFO:sockeye.training:__call__] E=33 B=135750 s/sec=1951.52 tok/sec=101710.96 u/sec=22.67 ppl=7.236512 [2023-12-17:12:59:39:INFO:sockeye.training:__call__] E=33 B=135800 s/sec=2096.07 tok/sec=97341.41 u/sec=23.44 ppl=7.242467 [2023-12-17:12:59:42:INFO:sockeye.training:__call__] E=34 B=135850 s/sec=2001.38 tok/sec=96402.73 u/sec=21.91 ppl=7.240988 [2023-12-17:12:59:44:INFO:sockeye.training:__call__] E=34 B=135900 s/sec=1920.30 tok/sec=103230.46 u/sec=22.73 ppl=7.229168 [2023-12-17:12:59:46:INFO:sockeye.training:__call__] E=34 B=135950 s/sec=2132.43 tok/sec=98956.65 u/sec=23.30 ppl=7.218418 [2023-12-17:12:59:48:INFO:sockeye.training:__call__] E=34 B=136000 s/sec=2077.02 tok/sec=100530.68 u/sec=23.26 ppl=7.208355 [2023-12-17:12:59:48:INFO:sockeye.training:_create_checkpoint] Checkpoint [34] Updates=136000 Epoch=34 Samples=12120552 Time-cost=172.245 Updates/sec=23.223 [2023-12-17:12:59:48:INFO:sockeye.training:_create_checkpoint] Checkpoint [34] Train-ppl=7.208355 [2023-12-17:12:59:51:INFO:sockeye.training:_evaluate] Checkpoint [34] Validation-ppl=30.158287 Validation-bleu=0.174623 Validation-chrf=32.169727 Validation-rouge1=0.175884 Validation-rouge2=0.084897 Validation-rougel=0.168130 Validation-length-ratio-mse=1.297719 Validation-ter=95.790343 Validation-avg-sec-per-sent=0.003459 Validation-decode-walltime=1.729492 [2023-12-17:12:59:51:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 10 checkpoints, best so far: 32.780032 [2023-12-17:12:59:51:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:12:59:51:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [34] Learning-rate=0.000146 [2023-12-17:12:59:51:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:12:59:51:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:12:59:51:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00034" [2023-12-17:12:59:54:INFO:sockeye.training:__call__] E=34 B=136050 s/sec=813.14 tok/sec=37892.84 u/sec=8.98 ppl=6.520297 [2023-12-17:12:59:56:INFO:sockeye.training:__call__] E=34 B=136100 s/sec=2133.33 tok/sec=95354.76 u/sec=23.72 ppl=6.529392 [2023-12-17:12:59:58:INFO:sockeye.training:__call__] E=34 B=136150 s/sec=1937.03 tok/sec=107240.58 u/sec=21.89 ppl=6.518262 [2023-12-17:13:00:00:INFO:sockeye.training:__call__] E=34 B=136200 s/sec=2151.66 tok/sec=98773.81 u/sec=23.47 ppl=6.528977 [2023-12-17:13:00:02:INFO:sockeye.training:__call__] E=34 B=136250 s/sec=1982.14 tok/sec=99456.54 u/sec=22.77 ppl=6.526840 [2023-12-17:13:00:04:INFO:sockeye.training:__call__] E=34 B=136300 s/sec=1711.89 tok/sec=103720.92 u/sec=22.81 ppl=6.499696 [2023-12-17:13:00:07:INFO:sockeye.training:__call__] E=34 B=136350 s/sec=2241.06 tok/sec=93142.12 u/sec=23.98 ppl=6.532026 [2023-12-17:13:00:09:INFO:sockeye.training:__call__] E=34 B=136400 s/sec=2195.00 tok/sec=96967.11 u/sec=23.25 ppl=6.549340 [2023-12-17:13:00:11:INFO:sockeye.training:__call__] E=34 B=136450 s/sec=2046.37 tok/sec=101076.54 u/sec=22.96 ppl=6.559263 [2023-12-17:13:00:13:INFO:sockeye.training:__call__] E=34 B=136500 s/sec=2304.39 tok/sec=94629.54 u/sec=23.96 ppl=6.569294 [2023-12-17:13:00:15:INFO:sockeye.training:__call__] E=34 B=136550 s/sec=1911.14 tok/sec=101646.34 u/sec=22.71 ppl=6.574305 [2023-12-17:13:00:17:INFO:sockeye.training:__call__] E=34 B=136600 s/sec=2110.72 tok/sec=98735.00 u/sec=23.43 ppl=6.581764 [2023-12-17:13:00:19:INFO:sockeye.training:__call__] E=34 B=136650 s/sec=2030.22 tok/sec=101109.32 u/sec=23.24 ppl=6.589138 [2023-12-17:13:00:22:INFO:sockeye.training:__call__] E=34 B=136700 s/sec=1924.45 tok/sec=102247.73 u/sec=22.82 ppl=6.589911 [2023-12-17:13:00:24:INFO:sockeye.training:__call__] E=34 B=136750 s/sec=2162.89 tok/sec=97782.18 u/sec=23.35 ppl=6.605959 [2023-12-17:13:00:26:INFO:sockeye.training:__call__] E=34 B=136800 s/sec=2056.51 tok/sec=94468.98 u/sec=24.02 ppl=6.618749 [2023-12-17:13:00:28:INFO:sockeye.training:__call__] E=34 B=136850 s/sec=2196.48 tok/sec=98325.80 u/sec=23.03 ppl=6.635925 [2023-12-17:13:00:30:INFO:sockeye.training:__call__] E=34 B=136900 s/sec=2098.96 tok/sec=96951.57 u/sec=23.55 ppl=6.644747 [2023-12-17:13:00:32:INFO:sockeye.training:__call__] E=34 B=136950 s/sec=2104.04 tok/sec=98222.87 u/sec=23.03 ppl=6.656243 [2023-12-17:13:00:35:INFO:sockeye.training:__call__] E=34 B=137000 s/sec=1974.32 tok/sec=99747.51 u/sec=23.02 ppl=6.661831 [2023-12-17:13:00:37:INFO:sockeye.training:__call__] E=34 B=137050 s/sec=1963.86 tok/sec=101532.29 u/sec=22.99 ppl=6.667812 [2023-12-17:13:00:39:INFO:sockeye.training:__call__] E=34 B=137100 s/sec=2068.61 tok/sec=99469.59 u/sec=22.96 ppl=6.677125 [2023-12-17:13:00:41:INFO:sockeye.training:__call__] E=34 B=137150 s/sec=2162.80 tok/sec=96340.29 u/sec=22.99 ppl=6.691556 [2023-12-17:13:00:43:INFO:sockeye.training:__call__] E=34 B=137200 s/sec=2076.71 tok/sec=99474.25 u/sec=23.18 ppl=6.697477 [2023-12-17:13:00:45:INFO:sockeye.training:__call__] E=34 B=137250 s/sec=1930.51 tok/sec=100416.32 u/sec=23.11 ppl=6.702583 [2023-12-17:13:00:48:INFO:sockeye.training:__call__] E=34 B=137300 s/sec=2130.17 tok/sec=99851.61 u/sec=23.11 ppl=6.710946 [2023-12-17:13:00:50:INFO:sockeye.training:__call__] E=34 B=137350 s/sec=1894.85 tok/sec=101543.38 u/sec=22.73 ppl=6.716456 [2023-12-17:13:00:52:INFO:sockeye.training:__call__] E=34 B=137400 s/sec=2115.46 tok/sec=100873.36 u/sec=22.72 ppl=6.724599 [2023-12-17:13:00:54:INFO:sockeye.training:__call__] E=34 B=137450 s/sec=1850.80 tok/sec=101057.96 u/sec=23.23 ppl=6.725272 [2023-12-17:13:00:56:INFO:sockeye.training:__call__] E=34 B=137500 s/sec=1880.72 tok/sec=101477.96 u/sec=22.91 ppl=6.732438 [2023-12-17:13:00:58:INFO:sockeye.training:__call__] E=34 B=137550 s/sec=1973.38 tok/sec=96201.45 u/sec=23.58 ppl=6.738172 [2023-12-17:13:01:01:INFO:sockeye.training:__call__] E=34 B=137600 s/sec=1983.84 tok/sec=98879.79 u/sec=23.22 ppl=6.742607 [2023-12-17:13:01:03:INFO:sockeye.training:__call__] E=34 B=137650 s/sec=2023.14 tok/sec=99163.62 u/sec=23.24 ppl=6.750508 [2023-12-17:13:01:05:INFO:sockeye.training:__call__] E=34 B=137700 s/sec=1839.26 tok/sec=104043.25 u/sec=22.06 ppl=6.758937 [2023-12-17:13:01:07:INFO:sockeye.training:__call__] E=34 B=137750 s/sec=2022.88 tok/sec=98991.69 u/sec=23.11 ppl=6.765262 [2023-12-17:13:01:09:INFO:sockeye.training:__call__] E=34 B=137800 s/sec=2196.13 tok/sec=96419.32 u/sec=23.50 ppl=6.774126 [2023-12-17:13:01:11:INFO:sockeye.training:__call__] E=34 B=137850 s/sec=2029.58 tok/sec=99739.29 u/sec=22.94 ppl=6.781237 [2023-12-17:13:01:14:INFO:sockeye.training:__call__] E=34 B=137900 s/sec=2270.01 tok/sec=96813.67 u/sec=23.37 ppl=6.791395 [2023-12-17:13:01:16:INFO:sockeye.training:__call__] E=34 B=137950 s/sec=1958.97 tok/sec=102113.80 u/sec=22.72 ppl=6.796189 [2023-12-17:13:01:18:INFO:sockeye.training:__call__] E=34 B=138000 s/sec=2072.10 tok/sec=102372.52 u/sec=22.52 ppl=6.802502 [2023-12-17:13:01:20:INFO:sockeye.training:__call__] E=34 B=138050 s/sec=2115.57 tok/sec=97756.28 u/sec=23.32 ppl=6.812168 [2023-12-17:13:01:22:INFO:sockeye.training:__call__] E=34 B=138100 s/sec=2142.66 tok/sec=95924.84 u/sec=23.62 ppl=6.820795 [2023-12-17:13:01:24:INFO:sockeye.training:__call__] E=34 B=138150 s/sec=2243.39 tok/sec=96488.44 u/sec=23.45 ppl=6.825976 [2023-12-17:13:01:27:INFO:sockeye.training:__call__] E=34 B=138200 s/sec=2036.36 tok/sec=99774.16 u/sec=23.31 ppl=6.833162 [2023-12-17:13:01:29:INFO:sockeye.training:__call__] E=34 B=138250 s/sec=1861.68 tok/sec=101397.86 u/sec=22.77 ppl=6.840238 [2023-12-17:13:01:31:INFO:sockeye.training:__call__] E=34 B=138300 s/sec=2109.46 tok/sec=95899.48 u/sec=23.50 ppl=6.846689 [2023-12-17:13:01:33:INFO:sockeye.training:__call__] E=34 B=138350 s/sec=2343.02 tok/sec=93659.34 u/sec=24.05 ppl=6.854399 [2023-12-17:13:01:35:INFO:sockeye.training:__call__] E=34 B=138400 s/sec=1879.94 tok/sec=102407.90 u/sec=22.64 ppl=6.856661 [2023-12-17:13:01:37:INFO:sockeye.training:__call__] E=34 B=138450 s/sec=2140.70 tok/sec=97547.82 u/sec=23.11 ppl=6.864717 [2023-12-17:13:01:40:INFO:sockeye.training:__call__] E=34 B=138500 s/sec=1763.45 tok/sec=101972.53 u/sec=22.87 ppl=6.867440 [2023-12-17:13:01:42:INFO:sockeye.training:__call__] E=34 B=138550 s/sec=2238.84 tok/sec=93877.05 u/sec=24.13 ppl=6.875731 [2023-12-17:13:01:44:INFO:sockeye.training:__call__] E=34 B=138600 s/sec=1936.78 tok/sec=101920.07 u/sec=23.01 ppl=6.881959 [2023-12-17:13:01:46:INFO:sockeye.training:__call__] E=34 B=138650 s/sec=2007.21 tok/sec=101364.24 u/sec=23.06 ppl=6.888349 [2023-12-17:13:01:48:INFO:sockeye.training:__call__] E=34 B=138700 s/sec=2218.33 tok/sec=95139.94 u/sec=23.50 ppl=6.896233 [2023-12-17:13:01:50:INFO:sockeye.training:__call__] E=34 B=138750 s/sec=2210.24 tok/sec=96479.36 u/sec=23.18 ppl=6.904839 [2023-12-17:13:01:52:INFO:sockeye.training:__call__] E=34 B=138800 s/sec=2119.97 tok/sec=100391.53 u/sec=22.73 ppl=6.911518 [2023-12-17:13:01:55:INFO:sockeye.training:__call__] E=34 B=138850 s/sec=2230.22 tok/sec=98249.45 u/sec=23.39 ppl=6.920710 [2023-12-17:13:01:57:INFO:sockeye.training:__call__] E=34 B=138900 s/sec=1942.13 tok/sec=103967.75 u/sec=22.69 ppl=6.923752 [2023-12-17:13:01:59:INFO:sockeye.training:__call__] E=34 B=138950 s/sec=2221.59 tok/sec=96428.43 u/sec=23.73 ppl=6.933174 [2023-12-17:13:02:01:INFO:sockeye.training:__call__] E=34 B=139000 s/sec=1970.75 tok/sec=102110.81 u/sec=23.02 ppl=6.938540 [2023-12-17:13:02:03:INFO:sockeye.training:__call__] E=34 B=139050 s/sec=2235.68 tok/sec=97429.42 u/sec=23.33 ppl=6.945179 [2023-12-17:13:02:05:INFO:sockeye.training:__call__] E=34 B=139100 s/sec=2042.66 tok/sec=99508.79 u/sec=23.30 ppl=6.947931 [2023-12-17:13:02:07:INFO:sockeye.training:__call__] E=34 B=139150 s/sec=2135.83 tok/sec=97808.40 u/sec=23.30 ppl=6.953195 [2023-12-17:13:02:10:INFO:sockeye.training:__call__] E=34 B=139200 s/sec=1960.56 tok/sec=98806.12 u/sec=22.95 ppl=6.959543 [2023-12-17:13:02:12:INFO:sockeye.training:__call__] E=34 B=139250 s/sec=2108.15 tok/sec=98690.48 u/sec=22.72 ppl=6.964947 [2023-12-17:13:02:14:INFO:sockeye.training:__call__] E=34 B=139300 s/sec=2218.81 tok/sec=94780.68 u/sec=23.79 ppl=6.971198 [2023-12-17:13:02:16:INFO:sockeye.training:__call__] E=34 B=139350 s/sec=1981.64 tok/sec=100334.01 u/sec=23.15 ppl=6.975912 [2023-12-17:13:02:18:INFO:sockeye.training:__call__] E=34 B=139400 s/sec=2235.62 tok/sec=98128.72 u/sec=23.29 ppl=6.981174 [2023-12-17:13:02:20:INFO:sockeye.training:__call__] E=34 B=139450 s/sec=2155.57 tok/sec=97389.81 u/sec=23.39 ppl=6.987570 [2023-12-17:13:02:23:INFO:sockeye.training:__call__] E=34 B=139500 s/sec=2259.78 tok/sec=96843.41 u/sec=23.23 ppl=6.994605 [2023-12-17:13:02:25:INFO:sockeye.training:__call__] E=34 B=139550 s/sec=1924.26 tok/sec=104141.50 u/sec=22.61 ppl=6.999259 [2023-12-17:13:02:27:INFO:sockeye.training:__call__] E=34 B=139600 s/sec=2078.55 tok/sec=97088.61 u/sec=23.28 ppl=7.004989 [2023-12-17:13:02:29:INFO:sockeye.training:__call__] E=34 B=139650 s/sec=2148.02 tok/sec=98496.62 u/sec=23.23 ppl=7.010396 [2023-12-17:13:02:31:INFO:sockeye.training:__call__] E=34 B=139700 s/sec=2178.95 tok/sec=94966.68 u/sec=24.02 ppl=7.014914 [2023-12-17:13:02:33:INFO:sockeye.training:__call__] E=34 B=139750 s/sec=2151.15 tok/sec=95092.33 u/sec=23.80 ppl=7.021917 [2023-12-17:13:02:36:INFO:sockeye.training:__call__] E=34 B=139800 s/sec=1724.55 tok/sec=105331.05 u/sec=22.00 ppl=7.024543 [2023-12-17:13:02:38:INFO:sockeye.training:__call__] E=35 B=139850 s/sec=1880.92 tok/sec=96971.69 u/sec=21.77 ppl=7.021213 [2023-12-17:13:02:40:INFO:sockeye.training:__call__] E=35 B=139900 s/sec=1882.24 tok/sec=100464.40 u/sec=22.98 ppl=7.011102 [2023-12-17:13:02:42:INFO:sockeye.training:__call__] E=35 B=139950 s/sec=2160.49 tok/sec=101091.70 u/sec=22.93 ppl=7.001555 [2023-12-17:13:02:44:INFO:sockeye.training:__call__] E=35 B=140000 s/sec=2054.01 tok/sec=98800.99 u/sec=23.26 ppl=6.993155 [2023-12-17:13:02:44:INFO:sockeye.training:_create_checkpoint] Checkpoint [35] Updates=140000 Epoch=35 Samples=12476928 Time-cost=172.880 Updates/sec=23.137 [2023-12-17:13:02:44:INFO:sockeye.training:_create_checkpoint] Checkpoint [35] Train-ppl=6.993155 [2023-12-17:13:02:47:INFO:sockeye.training:_evaluate] Checkpoint [35] Validation-ppl=30.597186 Validation-bleu=0.174329 Validation-chrf=31.597160 Validation-rouge1=0.167264 Validation-rouge2=0.083191 Validation-rougel=0.160078 Validation-length-ratio-mse=1.287048 Validation-ter=96.698308 Validation-avg-sec-per-sent=0.003397 Validation-decode-walltime=1.698481 [2023-12-17:13:02:47:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 11 checkpoints, best so far: 32.780032 [2023-12-17:13:02:47:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:13:02:47:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [35] Learning-rate=0.000146 [2023-12-17:13:02:47:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:13:02:47:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=3, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:13:02:48:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00035" [2023-12-17:13:02:50:INFO:sockeye.training:__call__] E=35 B=140050 s/sec=778.09 tok/sec=41122.40 u/sec=8.99 ppl=6.258254 [2023-12-17:13:02:52:INFO:sockeye.training:__call__] E=35 B=140100 s/sec=2121.38 tok/sec=96541.85 u/sec=23.59 ppl=6.278848 [2023-12-17:13:02:54:INFO:sockeye.training:__call__] E=35 B=140150 s/sec=2095.84 tok/sec=100483.12 u/sec=22.90 ppl=6.322257 [2023-12-17:13:02:56:INFO:sockeye.training:__call__] E=35 B=140200 s/sec=1858.01 tok/sec=100516.74 u/sec=23.00 ppl=6.323372 [2023-12-17:13:02:58:INFO:sockeye.training:__call__] E=35 B=140250 s/sec=2073.29 tok/sec=98095.40 u/sec=23.65 ppl=6.350065 [2023-12-17:13:03:01:INFO:sockeye.training:__call__] E=35 B=140300 s/sec=2081.76 tok/sec=96290.69 u/sec=23.66 ppl=6.367389 [2023-12-17:13:03:03:INFO:sockeye.training:__call__] E=35 B=140350 s/sec=1984.95 tok/sec=99579.13 u/sec=23.02 ppl=6.380817 [2023-12-17:13:03:05:INFO:sockeye.training:__call__] E=35 B=140400 s/sec=2146.28 tok/sec=98544.97 u/sec=23.01 ppl=6.399889 [2023-12-17:13:03:07:INFO:sockeye.training:__call__] E=35 B=140450 s/sec=2356.86 tok/sec=95364.00 u/sec=23.64 ppl=6.414419 [2023-12-17:13:03:09:INFO:sockeye.training:__call__] E=35 B=140500 s/sec=1941.25 tok/sec=101680.20 u/sec=22.98 ppl=6.411106 [2023-12-17:13:03:11:INFO:sockeye.training:__call__] E=35 B=140550 s/sec=1928.53 tok/sec=101389.60 u/sec=22.74 ppl=6.412377 [2023-12-17:13:03:14:INFO:sockeye.training:__call__] E=35 B=140600 s/sec=2114.72 tok/sec=99803.05 u/sec=22.95 ppl=6.421554 [2023-12-17:13:03:16:INFO:sockeye.training:__call__] E=35 B=140650 s/sec=2045.95 tok/sec=99416.85 u/sec=23.08 ppl=6.430500 [2023-12-17:13:03:18:INFO:sockeye.training:__call__] E=35 B=140700 s/sec=2180.76 tok/sec=97336.94 u/sec=23.50 ppl=6.437279 [2023-12-17:13:03:20:INFO:sockeye.training:__call__] E=35 B=140750 s/sec=2286.89 tok/sec=97994.45 u/sec=23.47 ppl=6.456258 [2023-12-17:13:03:22:INFO:sockeye.training:__call__] E=35 B=140800 s/sec=2069.18 tok/sec=100498.70 u/sec=23.01 ppl=6.466013 [2023-12-17:13:03:24:INFO:sockeye.training:__call__] E=35 B=140850 s/sec=1965.91 tok/sec=99102.33 u/sec=22.71 ppl=6.475937 [2023-12-17:13:03:27:INFO:sockeye.training:__call__] E=35 B=140900 s/sec=1832.01 tok/sec=103020.97 u/sec=22.32 ppl=6.479906 [2023-12-17:13:03:29:INFO:sockeye.training:__call__] E=35 B=140950 s/sec=1872.49 tok/sec=102231.31 u/sec=22.81 ppl=6.487321 [2023-12-17:13:03:31:INFO:sockeye.training:__call__] E=35 B=141000 s/sec=2045.24 tok/sec=98467.96 u/sec=23.16 ppl=6.493984 [2023-12-17:13:03:33:INFO:sockeye.training:__call__] E=35 B=141050 s/sec=2031.48 tok/sec=100209.78 u/sec=22.92 ppl=6.503850 [2023-12-17:13:03:35:INFO:sockeye.training:__call__] E=35 B=141100 s/sec=2126.30 tok/sec=98325.41 u/sec=23.36 ppl=6.511176 [2023-12-17:13:03:37:INFO:sockeye.training:__call__] E=35 B=141150 s/sec=2136.51 tok/sec=96207.24 u/sec=23.72 ppl=6.525775 [2023-12-17:13:03:40:INFO:sockeye.training:__call__] E=35 B=141200 s/sec=2154.51 tok/sec=98928.08 u/sec=23.38 ppl=6.534334 [2023-12-17:13:03:42:INFO:sockeye.training:__call__] E=35 B=141250 s/sec=2363.87 tok/sec=93265.26 u/sec=23.98 ppl=6.550126 [2023-12-17:13:03:44:INFO:sockeye.training:__call__] E=35 B=141300 s/sec=1970.40 tok/sec=99441.02 u/sec=23.59 ppl=6.551810 [2023-12-17:13:03:46:INFO:sockeye.training:__call__] E=35 B=141350 s/sec=2291.45 tok/sec=94778.27 u/sec=23.44 ppl=6.562912 [2023-12-17:13:03:48:INFO:sockeye.training:__call__] E=35 B=141400 s/sec=2116.28 tok/sec=97242.46 u/sec=23.70 ppl=6.571601 [2023-12-17:13:03:50:INFO:sockeye.training:__call__] E=35 B=141450 s/sec=1940.07 tok/sec=101268.57 u/sec=23.14 ppl=6.579787 [2023-12-17:13:03:52:INFO:sockeye.training:__call__] E=35 B=141500 s/sec=2030.33 tok/sec=99705.14 u/sec=22.82 ppl=6.586924 [2023-12-17:13:03:54:INFO:sockeye.training:__call__] E=35 B=141550 s/sec=2279.43 tok/sec=96962.65 u/sec=23.67 ppl=6.597755 [2023-12-17:13:03:57:INFO:sockeye.training:__call__] E=35 B=141600 s/sec=2015.53 tok/sec=101302.34 u/sec=22.82 ppl=6.602822 [2023-12-17:13:03:59:INFO:sockeye.training:__call__] E=35 B=141650 s/sec=1793.45 tok/sec=103360.86 u/sec=22.64 ppl=6.606318 [2023-12-17:13:04:01:INFO:sockeye.training:__call__] E=35 B=141700 s/sec=1986.10 tok/sec=99155.64 u/sec=23.33 ppl=6.611344 [2023-12-17:13:04:03:INFO:sockeye.training:__call__] E=35 B=141750 s/sec=2184.72 tok/sec=93805.55 u/sec=23.87 ppl=6.620325 [2023-12-17:13:04:05:INFO:sockeye.training:__call__] E=35 B=141800 s/sec=2126.09 tok/sec=100686.00 u/sec=22.71 ppl=6.624357 [2023-12-17:13:04:07:INFO:sockeye.training:__call__] E=35 B=141850 s/sec=2231.77 tok/sec=96364.33 u/sec=23.48 ppl=6.631917 [2023-12-17:13:04:10:INFO:sockeye.training:__call__] E=35 B=141900 s/sec=2065.71 tok/sec=98740.88 u/sec=23.05 ppl=6.637748 [2023-12-17:13:04:12:INFO:sockeye.training:__call__] E=35 B=141950 s/sec=2001.61 tok/sec=97768.99 u/sec=23.60 ppl=6.642443 [2023-12-17:13:04:14:INFO:sockeye.training:__call__] E=35 B=142000 s/sec=2107.12 tok/sec=95703.06 u/sec=23.48 ppl=6.650041 [2023-12-17:13:04:16:INFO:sockeye.training:__call__] E=35 B=142050 s/sec=2243.95 tok/sec=96773.36 u/sec=23.03 ppl=6.659436 [2023-12-17:13:04:18:INFO:sockeye.training:__call__] E=35 B=142100 s/sec=1936.88 tok/sec=99695.42 u/sec=23.15 ppl=6.668565 [2023-12-17:13:04:20:INFO:sockeye.training:__call__] E=35 B=142150 s/sec=2353.05 tok/sec=95581.88 u/sec=23.76 ppl=6.677813 [2023-12-17:13:04:22:INFO:sockeye.training:__call__] E=35 B=142200 s/sec=1954.79 tok/sec=102132.96 u/sec=22.92 ppl=6.680785 [2023-12-17:13:04:25:INFO:sockeye.training:__call__] E=35 B=142250 s/sec=1766.43 tok/sec=103061.37 u/sec=21.86 ppl=6.681506 [2023-12-17:13:04:27:INFO:sockeye.training:__call__] E=35 B=142300 s/sec=1920.40 tok/sec=101795.61 u/sec=22.91 ppl=6.686783 [2023-12-17:13:04:29:INFO:sockeye.training:__call__] E=35 B=142350 s/sec=2156.96 tok/sec=96487.17 u/sec=23.53 ppl=6.694574 [2023-12-17:13:04:31:INFO:sockeye.training:__call__] E=35 B=142400 s/sec=2450.29 tok/sec=90852.18 u/sec=24.42 ppl=6.703117 [2023-12-17:13:04:33:INFO:sockeye.training:__call__] E=35 B=142450 s/sec=2028.51 tok/sec=99059.44 u/sec=23.18 ppl=6.706035 [2023-12-17:13:04:35:INFO:sockeye.training:__call__] E=35 B=142500 s/sec=1774.34 tok/sec=103043.31 u/sec=22.82 ppl=6.709621 [2023-12-17:13:04:38:INFO:sockeye.training:__call__] E=35 B=142550 s/sec=2019.05 tok/sec=100369.24 u/sec=23.07 ppl=6.713820 [2023-12-17:13:04:40:INFO:sockeye.training:__call__] E=35 B=142600 s/sec=2261.59 tok/sec=95934.16 u/sec=23.88 ppl=6.722011 [2023-12-17:13:04:42:INFO:sockeye.training:__call__] E=35 B=142650 s/sec=1805.69 tok/sec=104476.90 u/sec=22.62 ppl=6.725031 [2023-12-17:13:04:44:INFO:sockeye.training:__call__] E=35 B=142700 s/sec=2065.42 tok/sec=99523.84 u/sec=23.05 ppl=6.730202 [2023-12-17:13:04:46:INFO:sockeye.training:__call__] E=35 B=142750 s/sec=2170.58 tok/sec=98826.70 u/sec=23.27 ppl=6.735371 [2023-12-17:13:04:48:INFO:sockeye.training:__call__] E=35 B=142800 s/sec=2085.88 tok/sec=98632.47 u/sec=23.28 ppl=6.741952 [2023-12-17:13:04:51:INFO:sockeye.training:__call__] E=35 B=142850 s/sec=1880.36 tok/sec=103345.52 u/sec=23.13 ppl=6.746024 [2023-12-17:13:04:53:INFO:sockeye.training:__call__] E=35 B=142900 s/sec=2032.28 tok/sec=101326.32 u/sec=23.05 ppl=6.750380 [2023-12-17:13:04:55:INFO:sockeye.training:__call__] E=35 B=142950 s/sec=2127.67 tok/sec=99490.90 u/sec=23.41 ppl=6.754632 [2023-12-17:13:04:57:INFO:sockeye.training:__call__] E=35 B=143000 s/sec=1992.22 tok/sec=100281.18 u/sec=23.02 ppl=6.760008 [2023-12-17:13:04:59:INFO:sockeye.training:__call__] E=35 B=143050 s/sec=2008.82 tok/sec=100677.28 u/sec=23.08 ppl=6.764904 [2023-12-17:13:05:01:INFO:sockeye.training:__call__] E=35 B=143100 s/sec=2189.66 tok/sec=94831.86 u/sec=23.88 ppl=6.771350 [2023-12-17:13:05:03:INFO:sockeye.training:__call__] E=35 B=143150 s/sec=2053.39 tok/sec=97827.87 u/sec=22.96 ppl=6.776887 [2023-12-17:13:05:06:INFO:sockeye.training:__call__] E=35 B=143200 s/sec=2294.12 tok/sec=97279.60 u/sec=23.54 ppl=6.784014 [2023-12-17:13:05:08:INFO:sockeye.training:__call__] E=35 B=143250 s/sec=2121.97 tok/sec=98171.88 u/sec=23.39 ppl=6.789062 [2023-12-17:13:05:10:INFO:sockeye.training:__call__] E=35 B=143300 s/sec=2072.13 tok/sec=99051.36 u/sec=22.92 ppl=6.794418 [2023-12-17:13:05:12:INFO:sockeye.training:__call__] E=35 B=143350 s/sec=2270.56 tok/sec=95629.79 u/sec=23.77 ppl=6.801065 [2023-12-17:13:05:14:INFO:sockeye.training:__call__] E=35 B=143400 s/sec=2102.78 tok/sec=98487.17 u/sec=23.59 ppl=6.806920 [2023-12-17:13:05:16:INFO:sockeye.training:__call__] E=35 B=143450 s/sec=1855.48 tok/sec=105691.70 u/sec=22.13 ppl=6.810731 [2023-12-17:13:05:19:INFO:sockeye.training:__call__] E=35 B=143500 s/sec=2064.79 tok/sec=98809.46 u/sec=23.46 ppl=6.815717 [2023-12-17:13:05:21:INFO:sockeye.training:__call__] E=35 B=143550 s/sec=2277.99 tok/sec=95135.64 u/sec=23.77 ppl=6.820380 [2023-12-17:13:05:23:INFO:sockeye.training:__call__] E=35 B=143600 s/sec=1752.21 tok/sec=108644.31 u/sec=21.86 ppl=6.822991 [2023-12-17:13:05:25:INFO:sockeye.training:__call__] E=35 B=143650 s/sec=2061.00 tok/sec=99196.01 u/sec=23.25 ppl=6.827204 [2023-12-17:13:05:27:INFO:sockeye.training:__call__] E=35 B=143700 s/sec=2043.14 tok/sec=98219.35 u/sec=23.22 ppl=6.831806 [2023-12-17:13:05:29:INFO:sockeye.training:__call__] E=35 B=143750 s/sec=2142.04 tok/sec=99694.09 u/sec=23.24 ppl=6.836304 [2023-12-17:13:05:32:INFO:sockeye.training:__call__] E=35 B=143800 s/sec=2088.08 tok/sec=99488.66 u/sec=23.10 ppl=6.842585 [2023-12-17:13:05:34:INFO:sockeye.training:__call__] E=36 B=143850 s/sec=1968.36 tok/sec=92483.92 u/sec=22.61 ppl=6.839117 [2023-12-17:13:05:36:INFO:sockeye.training:__call__] E=36 B=143900 s/sec=2037.04 tok/sec=98854.66 u/sec=22.73 ppl=6.829451 [2023-12-17:13:05:38:INFO:sockeye.training:__call__] E=36 B=143950 s/sec=2120.85 tok/sec=98064.09 u/sec=23.54 ppl=6.820835 [2023-12-17:13:05:40:INFO:sockeye.training:__call__] E=36 B=144000 s/sec=2145.60 tok/sec=98431.28 u/sec=23.12 ppl=6.812174 [2023-12-17:13:05:40:INFO:sockeye.training:_create_checkpoint] Checkpoint [36] Updates=144000 Epoch=36 Samples=12833768 Time-cost=172.605 Updates/sec=23.174 [2023-12-17:13:05:40:INFO:sockeye.training:_create_checkpoint] Checkpoint [36] Train-ppl=6.812174 [2023-12-17:13:05:43:INFO:sockeye.training:_evaluate] Checkpoint [36] Validation-ppl=31.945002 Validation-bleu=0.165058 Validation-chrf=31.291654 Validation-rouge1=0.161237 Validation-rouge2=0.074632 Validation-rougel=0.154145 Validation-length-ratio-mse=1.273020 Validation-ter=96.120512 Validation-avg-sec-per-sent=0.003518 Validation-decode-walltime=1.758882 [2023-12-17:13:05:43:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 12 checkpoints, best so far: 32.780032 [2023-12-17:13:05:43:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:13:05:43:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [36] Learning-rate=0.000146 [2023-12-17:13:05:43:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:13:05:43:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=4, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:13:05:44:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00036" [2023-12-17:13:05:46:INFO:sockeye.training:__call__] E=36 B=144050 s/sec=758.54 tok/sec=39171.26 u/sec=8.88 ppl=6.156157 [2023-12-17:13:05:48:INFO:sockeye.training:__call__] E=36 B=144100 s/sec=2037.86 tok/sec=98559.64 u/sec=23.20 ppl=6.185276 [2023-12-17:13:05:50:INFO:sockeye.training:__call__] E=36 B=144150 s/sec=1867.96 tok/sec=101870.40 u/sec=22.49 ppl=6.167588 [2023-12-17:13:05:52:INFO:sockeye.training:__call__] E=36 B=144200 s/sec=2158.63 tok/sec=99400.87 u/sec=23.18 ppl=6.181231 [2023-12-17:13:05:55:INFO:sockeye.training:__call__] E=36 B=144250 s/sec=1832.99 tok/sec=100535.58 u/sec=22.91 ppl=6.173321 [2023-12-17:13:05:57:INFO:sockeye.training:__call__] E=36 B=144300 s/sec=2304.68 tok/sec=95214.37 u/sec=23.89 ppl=6.223812 [2023-12-17:13:05:59:INFO:sockeye.training:__call__] E=36 B=144350 s/sec=2224.95 tok/sec=99034.11 u/sec=23.06 ppl=6.243394 [2023-12-17:13:06:01:INFO:sockeye.training:__call__] E=36 B=144400 s/sec=1954.71 tok/sec=101820.64 u/sec=22.88 ppl=6.238646 [2023-12-17:13:06:03:INFO:sockeye.training:__call__] E=36 B=144450 s/sec=2167.75 tok/sec=99612.78 u/sec=23.12 ppl=6.238784 [2023-12-17:13:06:05:INFO:sockeye.training:__call__] E=36 B=144500 s/sec=2186.80 tok/sec=98070.35 u/sec=23.32 ppl=6.251659 [2023-12-17:13:06:08:INFO:sockeye.training:__call__] E=36 B=144550 s/sec=2006.59 tok/sec=100322.00 u/sec=22.84 ppl=6.246144 [2023-12-17:13:06:10:INFO:sockeye.training:__call__] E=36 B=144600 s/sec=2035.07 tok/sec=98907.92 u/sec=23.34 ppl=6.258827 [2023-12-17:13:06:12:INFO:sockeye.training:__call__] E=36 B=144650 s/sec=1957.33 tok/sec=100654.10 u/sec=23.17 ppl=6.264692 [2023-12-17:13:06:14:INFO:sockeye.training:__call__] E=36 B=144700 s/sec=2017.13 tok/sec=98421.24 u/sec=23.13 ppl=6.265950 [2023-12-17:13:06:16:INFO:sockeye.training:__call__] E=36 B=144750 s/sec=1973.68 tok/sec=102039.86 u/sec=22.55 ppl=6.276620 [2023-12-17:13:06:18:INFO:sockeye.training:__call__] E=36 B=144800 s/sec=2017.64 tok/sec=99144.00 u/sec=23.01 ppl=6.279135 [2023-12-17:13:06:21:INFO:sockeye.training:__call__] E=36 B=144850 s/sec=2233.10 tok/sec=94842.12 u/sec=23.82 ppl=6.297447 [2023-12-17:13:06:23:INFO:sockeye.training:__call__] E=36 B=144900 s/sec=2209.21 tok/sec=94259.49 u/sec=23.72 ppl=6.310557 [2023-12-17:13:06:25:INFO:sockeye.training:__call__] E=36 B=144950 s/sec=2144.09 tok/sec=99245.10 u/sec=22.95 ppl=6.321881 [2023-12-17:13:06:27:INFO:sockeye.training:__call__] E=36 B=145000 s/sec=2128.20 tok/sec=97717.22 u/sec=23.42 ppl=6.331375 [2023-12-17:13:06:29:INFO:sockeye.training:__call__] E=36 B=145050 s/sec=2275.49 tok/sec=96597.88 u/sec=23.43 ppl=6.345865 [2023-12-17:13:06:31:INFO:sockeye.training:__call__] E=36 B=145100 s/sec=2009.78 tok/sec=98689.05 u/sec=23.43 ppl=6.350082 [2023-12-17:13:06:33:INFO:sockeye.training:__call__] E=36 B=145150 s/sec=2051.09 tok/sec=98242.88 u/sec=23.35 ppl=6.357061 [2023-12-17:13:06:36:INFO:sockeye.training:__call__] E=36 B=145200 s/sec=1889.55 tok/sec=102548.66 u/sec=22.58 ppl=6.363937 [2023-12-17:13:06:38:INFO:sockeye.training:__call__] E=36 B=145250 s/sec=2106.36 tok/sec=97967.92 u/sec=23.34 ppl=6.378145 [2023-12-17:13:06:40:INFO:sockeye.training:__call__] E=36 B=145300 s/sec=2157.95 tok/sec=97293.21 u/sec=23.17 ppl=6.388750 [2023-12-17:13:06:42:INFO:sockeye.training:__call__] E=36 B=145350 s/sec=2214.13 tok/sec=97675.47 u/sec=23.34 ppl=6.396938 [2023-12-17:13:06:44:INFO:sockeye.training:__call__] E=36 B=145400 s/sec=1868.33 tok/sec=101775.35 u/sec=22.50 ppl=6.403127 [2023-12-17:13:06:46:INFO:sockeye.training:__call__] E=36 B=145450 s/sec=2118.82 tok/sec=99230.86 u/sec=23.27 ppl=6.410203 [2023-12-17:13:06:49:INFO:sockeye.training:__call__] E=36 B=145500 s/sec=2097.85 tok/sec=102386.62 u/sec=22.37 ppl=6.416993 [2023-12-17:13:06:51:INFO:sockeye.training:__call__] E=36 B=145550 s/sec=2022.55 tok/sec=99045.66 u/sec=23.24 ppl=6.423152 [2023-12-17:13:06:53:INFO:sockeye.training:__call__] E=36 B=145600 s/sec=1960.95 tok/sec=103475.32 u/sec=22.41 ppl=6.431179 [2023-12-17:13:06:55:INFO:sockeye.training:__call__] E=36 B=145650 s/sec=1983.25 tok/sec=99695.70 u/sec=22.83 ppl=6.439560 [2023-12-17:13:06:57:INFO:sockeye.training:__call__] E=36 B=145700 s/sec=2201.60 tok/sec=96930.71 u/sec=23.48 ppl=6.450180 [2023-12-17:13:06:59:INFO:sockeye.training:__call__] E=36 B=145750 s/sec=1953.83 tok/sec=99645.20 u/sec=23.13 ppl=6.455053 [2023-12-17:13:07:02:INFO:sockeye.training:__call__] E=36 B=145800 s/sec=1944.14 tok/sec=102731.45 u/sec=22.63 ppl=6.460812 [2023-12-17:13:07:04:INFO:sockeye.training:__call__] E=36 B=145850 s/sec=2118.35 tok/sec=96804.29 u/sec=23.52 ppl=6.468146 [2023-12-17:13:07:06:INFO:sockeye.training:__call__] E=36 B=145900 s/sec=2001.99 tok/sec=100445.01 u/sec=22.96 ppl=6.474177 [2023-12-17:13:07:08:INFO:sockeye.training:__call__] E=36 B=145950 s/sec=1819.50 tok/sec=100756.80 u/sec=22.74 ppl=6.477368 [2023-12-17:13:07:10:INFO:sockeye.training:__call__] E=36 B=146000 s/sec=2261.07 tok/sec=93859.31 u/sec=23.83 ppl=6.487441 [2023-12-17:13:07:12:INFO:sockeye.training:__call__] E=36 B=146050 s/sec=1799.28 tok/sec=102198.93 u/sec=22.95 ppl=6.490054 [2023-12-17:13:07:15:INFO:sockeye.training:__call__] E=36 B=146100 s/sec=2097.75 tok/sec=100514.88 u/sec=23.08 ppl=6.497113 [2023-12-17:13:07:17:INFO:sockeye.training:__call__] E=36 B=146150 s/sec=2004.72 tok/sec=100102.83 u/sec=23.12 ppl=6.504079 [2023-12-17:13:07:19:INFO:sockeye.training:__call__] E=36 B=146200 s/sec=1985.22 tok/sec=99968.84 u/sec=23.28 ppl=6.511512 [2023-12-17:13:07:21:INFO:sockeye.training:__call__] E=36 B=146250 s/sec=2196.67 tok/sec=96638.30 u/sec=23.55 ppl=6.519710 [2023-12-17:13:07:23:INFO:sockeye.training:__call__] E=36 B=146300 s/sec=2272.05 tok/sec=95937.58 u/sec=23.51 ppl=6.528350 [2023-12-17:13:07:25:INFO:sockeye.training:__call__] E=36 B=146350 s/sec=2058.92 tok/sec=100588.91 u/sec=23.31 ppl=6.532503 [2023-12-17:13:07:28:INFO:sockeye.training:__call__] E=36 B=146400 s/sec=2133.40 tok/sec=99519.26 u/sec=22.99 ppl=6.541593 [2023-12-17:13:07:30:INFO:sockeye.training:__call__] E=36 B=146450 s/sec=2157.01 tok/sec=98238.16 u/sec=23.12 ppl=6.548271 [2023-12-17:13:07:32:INFO:sockeye.training:__call__] E=36 B=146500 s/sec=2029.90 tok/sec=99972.69 u/sec=23.32 ppl=6.553356 [2023-12-17:13:07:34:INFO:sockeye.training:__call__] E=36 B=146550 s/sec=2189.45 tok/sec=99224.93 u/sec=23.51 ppl=6.558838 [2023-12-17:13:07:36:INFO:sockeye.training:__call__] E=36 B=146600 s/sec=2162.74 tok/sec=99544.93 u/sec=22.99 ppl=6.565025 [2023-12-17:13:07:38:INFO:sockeye.training:__call__] E=36 B=146650 s/sec=2422.76 tok/sec=94332.32 u/sec=23.70 ppl=6.574968 [2023-12-17:13:07:40:INFO:sockeye.training:__call__] E=36 B=146700 s/sec=2070.76 tok/sec=97757.83 u/sec=23.28 ppl=6.581144 [2023-12-17:13:07:43:INFO:sockeye.training:__call__] E=36 B=146750 s/sec=2144.16 tok/sec=97527.75 u/sec=23.15 ppl=6.586556 [2023-12-17:13:07:45:INFO:sockeye.training:__call__] E=36 B=146800 s/sec=1901.89 tok/sec=101189.32 u/sec=22.95 ppl=6.589838 [2023-12-17:13:07:47:INFO:sockeye.training:__call__] E=36 B=146850 s/sec=2101.53 tok/sec=99869.06 u/sec=23.08 ppl=6.595216 [2023-12-17:13:07:49:INFO:sockeye.training:__call__] E=36 B=146900 s/sec=2110.93 tok/sec=97808.95 u/sec=23.48 ppl=6.600367 [2023-12-17:13:07:51:INFO:sockeye.training:__call__] E=36 B=146950 s/sec=2106.73 tok/sec=97863.98 u/sec=22.94 ppl=6.605348 [2023-12-17:13:07:53:INFO:sockeye.training:__call__] E=36 B=147000 s/sec=2045.77 tok/sec=96466.34 u/sec=23.72 ppl=6.610571 [2023-12-17:13:07:55:INFO:sockeye.training:__call__] E=36 B=147050 s/sec=1972.46 tok/sec=100510.20 u/sec=23.13 ppl=6.616345 [2023-12-17:13:07:58:INFO:sockeye.training:__call__] E=36 B=147100 s/sec=1824.99 tok/sec=101012.66 u/sec=23.18 ppl=6.619268 [2023-12-17:13:08:00:INFO:sockeye.training:__call__] E=36 B=147150 s/sec=2137.78 tok/sec=97756.70 u/sec=23.28 ppl=6.626717 [2023-12-17:13:08:02:INFO:sockeye.training:__call__] E=36 B=147200 s/sec=1943.90 tok/sec=102302.89 u/sec=22.97 ppl=6.630296 [2023-12-17:13:08:04:INFO:sockeye.training:__call__] E=36 B=147250 s/sec=1973.19 tok/sec=102679.56 u/sec=23.05 ppl=6.634916 [2023-12-17:13:08:06:INFO:sockeye.training:__call__] E=36 B=147300 s/sec=2148.66 tok/sec=98061.05 u/sec=22.92 ppl=6.640551 [2023-12-17:13:08:08:INFO:sockeye.training:__call__] E=36 B=147350 s/sec=2060.29 tok/sec=99455.22 u/sec=23.08 ppl=6.644987 [2023-12-17:13:08:11:INFO:sockeye.training:__call__] E=36 B=147400 s/sec=1912.91 tok/sec=102463.19 u/sec=22.47 ppl=6.647919 [2023-12-17:13:08:13:INFO:sockeye.training:__call__] E=36 B=147450 s/sec=1768.24 tok/sec=104372.67 u/sec=22.24 ppl=6.652110 [2023-12-17:13:08:15:INFO:sockeye.training:__call__] E=36 B=147500 s/sec=2022.72 tok/sec=101403.84 u/sec=23.24 ppl=6.657902 [2023-12-17:13:08:17:INFO:sockeye.training:__call__] E=36 B=147550 s/sec=2071.09 tok/sec=97144.56 u/sec=23.62 ppl=6.662457 [2023-12-17:13:08:19:INFO:sockeye.training:__call__] E=36 B=147600 s/sec=2169.13 tok/sec=96241.06 u/sec=23.13 ppl=6.668286 [2023-12-17:13:08:21:INFO:sockeye.training:__call__] E=36 B=147650 s/sec=2243.92 tok/sec=95086.98 u/sec=23.93 ppl=6.672907 [2023-12-17:13:08:24:INFO:sockeye.training:__call__] E=36 B=147700 s/sec=2160.59 tok/sec=95370.48 u/sec=23.77 ppl=6.679366 [2023-12-17:13:08:26:INFO:sockeye.training:__call__] E=36 B=147750 s/sec=2083.43 tok/sec=97391.88 u/sec=23.46 ppl=6.683600 [2023-12-17:13:08:28:INFO:sockeye.training:__call__] E=36 B=147800 s/sec=1803.15 tok/sec=101772.84 u/sec=23.05 ppl=6.686893 [2023-12-17:13:08:30:INFO:sockeye.training:__call__] E=37 B=147850 s/sec=2077.60 tok/sec=95313.65 u/sec=22.23 ppl=6.682727 [2023-12-17:13:08:32:INFO:sockeye.training:__call__] E=37 B=147900 s/sec=2194.54 tok/sec=98754.35 u/sec=23.49 ppl=6.675163 [2023-12-17:13:08:34:INFO:sockeye.training:__call__] E=37 B=147950 s/sec=2427.90 tok/sec=89777.54 u/sec=24.40 ppl=6.669977 [2023-12-17:13:08:36:INFO:sockeye.training:__call__] E=37 B=148000 s/sec=1893.48 tok/sec=101648.51 u/sec=22.85 ppl=6.662224 [2023-12-17:13:08:36:INFO:sockeye.training:_create_checkpoint] Checkpoint [37] Updates=148000 Epoch=37 Samples=13190544 Time-cost=172.766 Updates/sec=23.153 [2023-12-17:13:08:36:INFO:sockeye.training:_create_checkpoint] Checkpoint [37] Train-ppl=6.662224 [2023-12-17:13:08:39:INFO:sockeye.training:_evaluate] Checkpoint [37] Validation-ppl=31.825074 Validation-bleu=0.160289 Validation-chrf=30.138325 Validation-rouge1=0.153954 Validation-rouge2=0.067648 Validation-rougel=0.147759 Validation-length-ratio-mse=1.267349 Validation-ter=96.574494 Validation-avg-sec-per-sent=0.003218 Validation-decode-walltime=1.609221 [2023-12-17:13:08:39:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 13 checkpoints, best so far: 32.780032 [2023-12-17:13:08:39:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:13:08:39:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [37] Learning-rate=0.000146 [2023-12-17:13:08:40:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:13:08:40:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=5, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:13:08:40:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00037" [2023-12-17:13:08:42:INFO:sockeye.training:__call__] E=37 B=148050 s/sec=755.60 tok/sec=40944.67 u/sec=9.06 ppl=5.988557 [2023-12-17:13:08:44:INFO:sockeye.training:__call__] E=37 B=148100 s/sec=2127.81 tok/sec=99927.18 u/sec=22.69 ppl=6.051318 [2023-12-17:13:08:46:INFO:sockeye.training:__call__] E=37 B=148150 s/sec=2008.49 tok/sec=100622.19 u/sec=22.87 ppl=6.047136 [2023-12-17:13:08:49:INFO:sockeye.training:__call__] E=37 B=148200 s/sec=1900.55 tok/sec=101499.81 u/sec=22.93 ppl=6.043289 [2023-12-17:13:08:51:INFO:sockeye.training:__call__] E=37 B=148250 s/sec=1915.67 tok/sec=98237.95 u/sec=23.38 ppl=6.049184 [2023-12-17:13:08:53:INFO:sockeye.training:__call__] E=37 B=148300 s/sec=2424.39 tok/sec=95690.59 u/sec=23.90 ppl=6.071187 [2023-12-17:13:08:55:INFO:sockeye.training:__call__] E=37 B=148350 s/sec=2415.95 tok/sec=93143.39 u/sec=24.16 ppl=6.106497 [2023-12-17:13:08:57:INFO:sockeye.training:__call__] E=37 B=148400 s/sec=1989.30 tok/sec=102439.95 u/sec=22.73 ppl=6.114837 [2023-12-17:13:08:59:INFO:sockeye.training:__call__] E=37 B=148450 s/sec=2061.72 tok/sec=97617.92 u/sec=23.34 ppl=6.128035 [2023-12-17:13:09:01:INFO:sockeye.training:__call__] E=37 B=148500 s/sec=2024.19 tok/sec=102568.70 u/sec=22.59 ppl=6.132176 [2023-12-17:13:09:04:INFO:sockeye.training:__call__] E=37 B=148550 s/sec=1853.06 tok/sec=99635.83 u/sec=22.75 ppl=6.141472 [2023-12-17:13:09:06:INFO:sockeye.training:__call__] E=37 B=148600 s/sec=2079.25 tok/sec=99024.65 u/sec=23.41 ppl=6.153055 [2023-12-17:13:09:08:INFO:sockeye.training:__call__] E=37 B=148650 s/sec=2175.57 tok/sec=97665.70 u/sec=23.69 ppl=6.166319 [2023-12-17:13:09:10:INFO:sockeye.training:__call__] E=37 B=148700 s/sec=2016.18 tok/sec=100145.67 u/sec=23.29 ppl=6.167729 [2023-12-17:13:09:12:INFO:sockeye.training:__call__] E=37 B=148750 s/sec=2059.78 tok/sec=99047.51 u/sec=23.20 ppl=6.176261 [2023-12-17:13:09:14:INFO:sockeye.training:__call__] E=37 B=148800 s/sec=2246.75 tok/sec=96844.91 u/sec=23.29 ppl=6.187746 [2023-12-17:13:09:17:INFO:sockeye.training:__call__] E=37 B=148850 s/sec=2168.12 tok/sec=103957.33 u/sec=21.93 ppl=6.200288 [2023-12-17:13:09:19:INFO:sockeye.training:__call__] E=37 B=148900 s/sec=2007.65 tok/sec=100264.40 u/sec=23.07 ppl=6.206257 [2023-12-17:13:09:21:INFO:sockeye.training:__call__] E=37 B=148950 s/sec=2073.06 tok/sec=102169.56 u/sec=22.61 ppl=6.211338 [2023-12-17:13:09:23:INFO:sockeye.training:__call__] E=37 B=149000 s/sec=2051.73 tok/sec=101076.31 u/sec=23.02 ppl=6.222663 [2023-12-17:13:09:25:INFO:sockeye.training:__call__] E=37 B=149050 s/sec=1798.97 tok/sec=103934.14 u/sec=22.67 ppl=6.223063 [2023-12-17:13:09:28:INFO:sockeye.training:__call__] E=37 B=149100 s/sec=1806.47 tok/sec=104490.59 u/sec=22.81 ppl=6.225546 [2023-12-17:13:09:30:INFO:sockeye.training:__call__] E=37 B=149150 s/sec=1958.32 tok/sec=99251.27 u/sec=23.05 ppl=6.237134 [2023-12-17:13:09:32:INFO:sockeye.training:__call__] E=37 B=149200 s/sec=2224.03 tok/sec=98412.27 u/sec=23.44 ppl=6.247454 [2023-12-17:13:09:34:INFO:sockeye.training:__call__] E=37 B=149250 s/sec=2151.28 tok/sec=97231.98 u/sec=23.67 ppl=6.255501 [2023-12-17:13:09:36:INFO:sockeye.training:__call__] E=37 B=149300 s/sec=1858.16 tok/sec=99679.45 u/sec=22.95 ppl=6.259985 [2023-12-17:13:09:38:INFO:sockeye.training:__call__] E=37 B=149350 s/sec=2036.43 tok/sec=99338.79 u/sec=23.44 ppl=6.270770 [2023-12-17:13:09:40:INFO:sockeye.training:__call__] E=37 B=149400 s/sec=2244.35 tok/sec=94392.79 u/sec=23.90 ppl=6.282257 [2023-12-17:13:09:43:INFO:sockeye.training:__call__] E=37 B=149450 s/sec=1955.87 tok/sec=101923.72 u/sec=22.76 ppl=6.290507 [2023-12-17:13:09:45:INFO:sockeye.training:__call__] E=37 B=149500 s/sec=2300.02 tok/sec=96147.54 u/sec=24.00 ppl=6.299549 [2023-12-17:13:09:47:INFO:sockeye.training:__call__] E=37 B=149550 s/sec=1956.60 tok/sec=99291.69 u/sec=23.07 ppl=6.305806 [2023-12-17:13:09:49:INFO:sockeye.training:__call__] E=37 B=149600 s/sec=2076.40 tok/sec=100143.58 u/sec=23.26 ppl=6.309709 [2023-12-17:13:09:51:INFO:sockeye.training:__call__] E=37 B=149650 s/sec=2250.18 tok/sec=96027.74 u/sec=23.64 ppl=6.317112 [2023-12-17:13:09:53:INFO:sockeye.training:__call__] E=37 B=149700 s/sec=2252.12 tok/sec=96589.27 u/sec=23.86 ppl=6.324897 [2023-12-17:13:09:55:INFO:sockeye.training:__call__] E=37 B=149750 s/sec=2090.61 tok/sec=99605.32 u/sec=23.25 ppl=6.330030 [2023-12-17:13:09:57:INFO:sockeye.training:__call__] E=37 B=149800 s/sec=1928.32 tok/sec=101272.32 u/sec=23.31 ppl=6.336519 [2023-12-17:13:10:00:INFO:sockeye.training:__call__] E=37 B=149850 s/sec=1879.19 tok/sec=102571.10 u/sec=23.12 ppl=6.341645 [2023-12-17:13:10:02:INFO:sockeye.training:__call__] E=37 B=149900 s/sec=2162.74 tok/sec=97054.22 u/sec=23.67 ppl=6.350870 [2023-12-17:13:10:04:INFO:sockeye.training:__call__] E=37 B=149950 s/sec=1972.21 tok/sec=101652.28 u/sec=22.74 ppl=6.355555 [2023-12-17:13:10:06:INFO:sockeye.training:__call__] E=37 B=150000 s/sec=2062.19 tok/sec=100811.77 u/sec=22.65 ppl=6.362810 [2023-12-17:13:10:08:INFO:sockeye.training:__call__] E=37 B=150050 s/sec=2149.56 tok/sec=98413.67 u/sec=23.12 ppl=6.368929 [2023-12-17:13:10:10:INFO:sockeye.training:__call__] E=37 B=150100 s/sec=2209.36 tok/sec=98004.48 u/sec=23.36 ppl=6.376596 [2023-12-17:13:10:13:INFO:sockeye.training:__call__] E=37 B=150150 s/sec=2101.28 tok/sec=97002.73 u/sec=23.37 ppl=6.382475 [2023-12-17:13:10:15:INFO:sockeye.training:__call__] E=37 B=150200 s/sec=1916.18 tok/sec=103575.81 u/sec=22.77 ppl=6.387451 [2023-12-17:13:10:17:INFO:sockeye.training:__call__] E=37 B=150250 s/sec=1965.27 tok/sec=101562.17 u/sec=22.96 ppl=6.393141 [2023-12-17:13:10:19:INFO:sockeye.training:__call__] E=37 B=150300 s/sec=1753.02 tok/sec=103744.08 u/sec=22.45 ppl=6.397177 [2023-12-17:13:10:21:INFO:sockeye.training:__call__] E=37 B=150350 s/sec=1914.57 tok/sec=100966.19 u/sec=22.92 ppl=6.401187 [2023-12-17:13:10:24:INFO:sockeye.training:__call__] E=37 B=150400 s/sec=2070.55 tok/sec=100892.02 u/sec=23.07 ppl=6.407309 [2023-12-17:13:10:26:INFO:sockeye.training:__call__] E=37 B=150450 s/sec=2149.28 tok/sec=97117.76 u/sec=23.16 ppl=6.411630 [2023-12-17:13:10:28:INFO:sockeye.training:__call__] E=37 B=150500 s/sec=1962.32 tok/sec=99365.27 u/sec=23.10 ppl=6.414666 [2023-12-17:13:10:30:INFO:sockeye.training:__call__] E=37 B=150550 s/sec=2403.60 tok/sec=94718.62 u/sec=23.69 ppl=6.423537 [2023-12-17:13:10:32:INFO:sockeye.training:__call__] E=37 B=150600 s/sec=1832.51 tok/sec=104624.92 u/sec=22.37 ppl=6.428201 [2023-12-17:13:10:34:INFO:sockeye.training:__call__] E=37 B=150650 s/sec=2040.81 tok/sec=100836.38 u/sec=22.94 ppl=6.434028 [2023-12-17:13:10:37:INFO:sockeye.training:__call__] E=37 B=150700 s/sec=1994.30 tok/sec=99951.24 u/sec=23.08 ppl=6.439023 [2023-12-17:13:10:39:INFO:sockeye.training:__call__] E=37 B=150750 s/sec=2108.45 tok/sec=98141.48 u/sec=23.24 ppl=6.445550 [2023-12-17:13:10:41:INFO:sockeye.training:__call__] E=37 B=150800 s/sec=1870.56 tok/sec=101515.68 u/sec=22.57 ppl=6.448960 [2023-12-17:13:10:43:INFO:sockeye.training:__call__] E=37 B=150850 s/sec=2102.66 tok/sec=99304.80 u/sec=23.06 ppl=6.452112 [2023-12-17:13:10:45:INFO:sockeye.training:__call__] E=37 B=150900 s/sec=2077.59 tok/sec=99369.27 u/sec=23.10 ppl=6.458119 [2023-12-17:13:10:47:INFO:sockeye.training:__call__] E=37 B=150950 s/sec=2137.93 tok/sec=99199.97 u/sec=23.04 ppl=6.464053 [2023-12-17:13:10:50:INFO:sockeye.training:__call__] E=37 B=151000 s/sec=1970.67 tok/sec=100475.44 u/sec=22.64 ppl=6.469190 [2023-12-17:13:10:52:INFO:sockeye.training:__call__] E=37 B=151050 s/sec=2199.95 tok/sec=94548.22 u/sec=23.91 ppl=6.475233 [2023-12-17:13:10:54:INFO:sockeye.training:__call__] E=37 B=151100 s/sec=1980.91 tok/sec=99007.81 u/sec=23.40 ppl=6.479659 [2023-12-17:13:10:56:INFO:sockeye.training:__call__] E=37 B=151150 s/sec=1988.16 tok/sec=95734.59 u/sec=23.67 ppl=6.484230 [2023-12-17:13:10:58:INFO:sockeye.training:__call__] E=37 B=151200 s/sec=2237.02 tok/sec=97327.36 u/sec=23.58 ppl=6.489573 [2023-12-17:13:11:00:INFO:sockeye.training:__call__] E=37 B=151250 s/sec=2173.27 tok/sec=96542.68 u/sec=23.54 ppl=6.494921 [2023-12-17:13:11:02:INFO:sockeye.training:__call__] E=37 B=151300 s/sec=1799.85 tok/sec=100165.73 u/sec=23.29 ppl=6.497680 [2023-12-17:13:11:05:INFO:sockeye.training:__call__] E=37 B=151350 s/sec=1881.19 tok/sec=101231.72 u/sec=22.96 ppl=6.501218 [2023-12-17:13:11:07:INFO:sockeye.training:__call__] E=37 B=151400 s/sec=2119.02 tok/sec=99030.13 u/sec=23.03 ppl=6.505892 [2023-12-17:13:11:09:INFO:sockeye.training:__call__] E=37 B=151450 s/sec=2208.37 tok/sec=98357.33 u/sec=23.51 ppl=6.511343 [2023-12-17:13:11:11:INFO:sockeye.training:__call__] E=37 B=151500 s/sec=2287.62 tok/sec=96139.91 u/sec=23.36 ppl=6.517208 [2023-12-17:13:11:13:INFO:sockeye.training:__call__] E=37 B=151550 s/sec=1912.40 tok/sec=102080.92 u/sec=22.51 ppl=6.520546 [2023-12-17:13:11:15:INFO:sockeye.training:__call__] E=37 B=151600 s/sec=2258.89 tok/sec=93610.68 u/sec=23.77 ppl=6.526154 [2023-12-17:13:11:17:INFO:sockeye.training:__call__] E=37 B=151650 s/sec=1934.08 tok/sec=100321.49 u/sec=23.02 ppl=6.530887 [2023-12-17:13:11:20:INFO:sockeye.training:__call__] E=37 B=151700 s/sec=2086.11 tok/sec=97779.41 u/sec=23.58 ppl=6.533884 [2023-12-17:13:11:22:INFO:sockeye.training:__call__] E=37 B=151750 s/sec=2137.90 tok/sec=97419.08 u/sec=23.48 ppl=6.538314 [2023-12-17:13:11:24:INFO:sockeye.training:__call__] E=37 B=151800 s/sec=2442.23 tok/sec=90036.00 u/sec=24.31 ppl=6.544926 [2023-12-17:13:11:26:INFO:sockeye.training:__call__] E=38 B=151850 s/sec=2100.68 tok/sec=94297.07 u/sec=22.10 ppl=6.540105 [2023-12-17:13:11:28:INFO:sockeye.training:__call__] E=38 B=151900 s/sec=2152.31 tok/sec=99101.81 u/sec=22.99 ppl=6.532515 [2023-12-17:13:11:30:INFO:sockeye.training:__call__] E=38 B=151950 s/sec=2157.73 tok/sec=96358.85 u/sec=23.09 ppl=6.525818 [2023-12-17:13:11:33:INFO:sockeye.training:__call__] E=38 B=152000 s/sec=1892.60 tok/sec=100870.21 u/sec=22.97 ppl=6.518390 [2023-12-17:13:11:33:INFO:sockeye.training:_create_checkpoint] Checkpoint [38] Updates=152000 Epoch=38 Samples=13546832 Time-cost=172.806 Updates/sec=23.147 [2023-12-17:13:11:33:INFO:sockeye.training:_create_checkpoint] Checkpoint [38] Train-ppl=6.518390 [2023-12-17:13:11:35:INFO:sockeye.training:_evaluate] Checkpoint [38] Validation-ppl=32.037354 Validation-bleu=0.148973 Validation-chrf=29.653748 Validation-rouge1=0.155914 Validation-rouge2=0.067473 Validation-rougel=0.148545 Validation-length-ratio-mse=1.349364 Validation-ter=100.288898 Validation-avg-sec-per-sent=0.003524 Validation-decode-walltime=1.761959 [2023-12-17:13:11:35:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 14 checkpoints, best so far: 32.780032 [2023-12-17:13:11:35:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:13:11:35:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [38] Learning-rate=0.000146 [2023-12-17:13:11:36:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:13:11:36:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=6, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:13:11:36:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00038" [2023-12-17:13:11:38:INFO:sockeye.training:__call__] E=38 B=152050 s/sec=793.84 tok/sec=39754.99 u/sec=8.96 ppl=5.990198 [2023-12-17:13:11:40:INFO:sockeye.training:__call__] E=38 B=152100 s/sec=1966.36 tok/sec=99873.66 u/sec=22.93 ppl=5.991153 [2023-12-17:13:11:42:INFO:sockeye.training:__call__] E=38 B=152150 s/sec=2238.26 tok/sec=95529.25 u/sec=23.43 ppl=6.015067 [2023-12-17:13:11:45:INFO:sockeye.training:__call__] E=38 B=152200 s/sec=2126.78 tok/sec=96783.61 u/sec=23.40 ppl=6.029489 [2023-12-17:13:11:47:INFO:sockeye.training:__call__] E=38 B=152250 s/sec=1989.64 tok/sec=100639.96 u/sec=23.20 ppl=6.046742 [2023-12-17:13:11:49:INFO:sockeye.training:__call__] E=38 B=152300 s/sec=2039.83 tok/sec=100095.45 u/sec=23.05 ppl=6.051387 [2023-12-17:13:11:51:INFO:sockeye.training:__call__] E=38 B=152350 s/sec=2205.03 tok/sec=97547.20 u/sec=23.48 ppl=6.062183 [2023-12-17:13:11:53:INFO:sockeye.training:__call__] E=38 B=152400 s/sec=1806.38 tok/sec=103287.58 u/sec=22.49 ppl=6.056680 [2023-12-17:13:11:55:INFO:sockeye.training:__call__] E=38 B=152450 s/sec=2075.21 tok/sec=98370.18 u/sec=23.62 ppl=6.054616 [2023-12-17:13:11:58:INFO:sockeye.training:__call__] E=38 B=152500 s/sec=1813.86 tok/sec=102219.87 u/sec=22.86 ppl=6.052842 [2023-12-17:13:12:00:INFO:sockeye.training:__call__] E=38 B=152550 s/sec=1914.92 tok/sec=100413.65 u/sec=22.97 ppl=6.059517 [2023-12-17:13:12:02:INFO:sockeye.training:__call__] E=38 B=152600 s/sec=1920.07 tok/sec=100923.89 u/sec=23.12 ppl=6.062629 [2023-12-17:13:12:04:INFO:sockeye.training:__call__] E=38 B=152650 s/sec=2206.76 tok/sec=96072.30 u/sec=23.90 ppl=6.078364 [2023-12-17:13:12:06:INFO:sockeye.training:__call__] E=38 B=152700 s/sec=1889.85 tok/sec=99980.12 u/sec=23.21 ppl=6.079116 [2023-12-17:13:12:08:INFO:sockeye.training:__call__] E=38 B=152750 s/sec=1834.25 tok/sec=101424.00 u/sec=22.97 ppl=6.078409 [2023-12-17:13:12:10:INFO:sockeye.training:__call__] E=38 B=152800 s/sec=2159.73 tok/sec=97656.10 u/sec=23.60 ppl=6.086129 [2023-12-17:13:12:13:INFO:sockeye.training:__call__] E=38 B=152850 s/sec=1970.31 tok/sec=102426.54 u/sec=22.97 ppl=6.091862 [2023-12-17:13:12:15:INFO:sockeye.training:__call__] E=38 B=152900 s/sec=2158.22 tok/sec=98559.95 u/sec=23.14 ppl=6.097931 [2023-12-17:13:12:17:INFO:sockeye.training:__call__] E=38 B=152950 s/sec=2166.07 tok/sec=99477.83 u/sec=22.95 ppl=6.108328 [2023-12-17:13:12:19:INFO:sockeye.training:__call__] E=38 B=153000 s/sec=2167.30 tok/sec=100167.54 u/sec=23.04 ppl=6.116980 [2023-12-17:13:12:21:INFO:sockeye.training:__call__] E=38 B=153050 s/sec=1937.21 tok/sec=102583.95 u/sec=22.93 ppl=6.117473 [2023-12-17:13:12:24:INFO:sockeye.training:__call__] E=38 B=153100 s/sec=2091.26 tok/sec=100467.34 u/sec=22.65 ppl=6.126290 [2023-12-17:13:12:26:INFO:sockeye.training:__call__] E=38 B=153150 s/sec=1864.59 tok/sec=104272.87 u/sec=22.50 ppl=6.131490 [2023-12-17:13:12:28:INFO:sockeye.training:__call__] E=38 B=153200 s/sec=2250.66 tok/sec=95896.47 u/sec=23.76 ppl=6.140835 [2023-12-17:13:12:30:INFO:sockeye.training:__call__] E=38 B=153250 s/sec=2100.81 tok/sec=98570.24 u/sec=23.32 ppl=6.149899 [2023-12-17:13:12:32:INFO:sockeye.training:__call__] E=38 B=153300 s/sec=2173.04 tok/sec=96970.66 u/sec=23.30 ppl=6.159085 [2023-12-17:13:12:34:INFO:sockeye.training:__call__] E=38 B=153350 s/sec=2112.00 tok/sec=97766.08 u/sec=23.70 ppl=6.165797 [2023-12-17:13:12:36:INFO:sockeye.training:__call__] E=38 B=153400 s/sec=2048.43 tok/sec=98415.84 u/sec=23.80 ppl=6.173840 [2023-12-17:13:12:39:INFO:sockeye.training:__call__] E=38 B=153450 s/sec=1972.36 tok/sec=99401.17 u/sec=23.53 ppl=6.178195 [2023-12-17:13:12:41:INFO:sockeye.training:__call__] E=38 B=153500 s/sec=2065.47 tok/sec=99557.84 u/sec=23.18 ppl=6.185679 [2023-12-17:13:12:43:INFO:sockeye.training:__call__] E=38 B=153550 s/sec=2132.22 tok/sec=99017.47 u/sec=23.02 ppl=6.194299 [2023-12-17:13:12:45:INFO:sockeye.training:__call__] E=38 B=153600 s/sec=2158.31 tok/sec=99364.29 u/sec=23.30 ppl=6.200011 [2023-12-17:13:12:47:INFO:sockeye.training:__call__] E=38 B=153650 s/sec=2088.73 tok/sec=98425.44 u/sec=23.11 ppl=6.207631 [2023-12-17:13:12:49:INFO:sockeye.training:__call__] E=38 B=153700 s/sec=2146.24 tok/sec=96569.63 u/sec=23.01 ppl=6.215494 [2023-12-17:13:12:51:INFO:sockeye.training:__call__] E=38 B=153750 s/sec=2364.23 tok/sec=96445.26 u/sec=23.64 ppl=6.224153 [2023-12-17:13:12:54:INFO:sockeye.training:__call__] E=38 B=153800 s/sec=2204.19 tok/sec=97736.58 u/sec=23.51 ppl=6.231516 [2023-12-17:13:12:56:INFO:sockeye.training:__call__] E=38 B=153850 s/sec=2176.31 tok/sec=95727.30 u/sec=23.70 ppl=6.238247 [2023-12-17:13:12:58:INFO:sockeye.training:__call__] E=38 B=153900 s/sec=2097.52 tok/sec=97336.93 u/sec=23.12 ppl=6.244979 [2023-12-17:13:13:00:INFO:sockeye.training:__call__] E=38 B=153950 s/sec=1736.20 tok/sec=102943.33 u/sec=22.33 ppl=6.247524 [2023-12-17:13:13:02:INFO:sockeye.training:__call__] E=38 B=154000 s/sec=2267.11 tok/sec=95984.44 u/sec=23.93 ppl=6.255939 [2023-12-17:13:13:04:INFO:sockeye.training:__call__] E=38 B=154050 s/sec=1851.39 tok/sec=99019.28 u/sec=23.33 ppl=6.261112 [2023-12-17:13:13:06:INFO:sockeye.training:__call__] E=38 B=154100 s/sec=2267.23 tok/sec=96815.09 u/sec=23.46 ppl=6.270112 [2023-12-17:13:13:09:INFO:sockeye.training:__call__] E=38 B=154150 s/sec=1921.50 tok/sec=100629.97 u/sec=23.18 ppl=6.274035 [2023-12-17:13:13:11:INFO:sockeye.training:__call__] E=38 B=154200 s/sec=1939.87 tok/sec=100531.79 u/sec=23.18 ppl=6.277202 [2023-12-17:13:13:13:INFO:sockeye.training:__call__] E=38 B=154250 s/sec=2196.15 tok/sec=96285.19 u/sec=23.46 ppl=6.286572 [2023-12-17:13:13:15:INFO:sockeye.training:__call__] E=38 B=154300 s/sec=2153.97 tok/sec=100086.94 u/sec=23.25 ppl=6.292031 [2023-12-17:13:13:17:INFO:sockeye.training:__call__] E=38 B=154350 s/sec=2124.00 tok/sec=99036.56 u/sec=23.01 ppl=6.297995 [2023-12-17:13:13:19:INFO:sockeye.training:__call__] E=38 B=154400 s/sec=2107.24 tok/sec=99485.64 u/sec=23.19 ppl=6.305432 [2023-12-17:13:13:22:INFO:sockeye.training:__call__] E=38 B=154450 s/sec=2048.16 tok/sec=98282.02 u/sec=23.06 ppl=6.310971 [2023-12-17:13:13:24:INFO:sockeye.training:__call__] E=38 B=154500 s/sec=2341.34 tok/sec=93221.69 u/sec=24.11 ppl=6.318797 [2023-12-17:13:13:26:INFO:sockeye.training:__call__] E=38 B=154550 s/sec=1875.34 tok/sec=103384.22 u/sec=22.41 ppl=6.323128 [2023-12-17:13:13:28:INFO:sockeye.training:__call__] E=38 B=154600 s/sec=1978.80 tok/sec=101516.52 u/sec=23.20 ppl=6.327103 [2023-12-17:13:13:30:INFO:sockeye.training:__call__] E=38 B=154650 s/sec=2040.62 tok/sec=101750.87 u/sec=23.02 ppl=6.332145 [2023-12-17:13:13:32:INFO:sockeye.training:__call__] E=38 B=154700 s/sec=2131.47 tok/sec=98517.74 u/sec=23.33 ppl=6.338938 [2023-12-17:13:13:34:INFO:sockeye.training:__call__] E=38 B=154750 s/sec=2156.93 tok/sec=97153.10 u/sec=23.73 ppl=6.344830 [2023-12-17:13:13:37:INFO:sockeye.training:__call__] E=38 B=154800 s/sec=2302.00 tok/sec=94210.86 u/sec=23.78 ppl=6.352431 [2023-12-17:13:13:39:INFO:sockeye.training:__call__] E=38 B=154850 s/sec=2245.15 tok/sec=98029.42 u/sec=23.19 ppl=6.358686 [2023-12-17:13:13:41:INFO:sockeye.training:__call__] E=38 B=154900 s/sec=2062.01 tok/sec=97150.38 u/sec=23.39 ppl=6.362779 [2023-12-17:13:13:43:INFO:sockeye.training:__call__] E=38 B=154950 s/sec=1945.61 tok/sec=100221.20 u/sec=23.21 ppl=6.366304 [2023-12-17:13:13:45:INFO:sockeye.training:__call__] E=38 B=155000 s/sec=1852.25 tok/sec=100720.91 u/sec=23.25 ppl=6.368048 [2023-12-17:13:13:47:INFO:sockeye.training:__call__] E=38 B=155050 s/sec=2136.69 tok/sec=97075.64 u/sec=23.68 ppl=6.373438 [2023-12-17:13:13:49:INFO:sockeye.training:__call__] E=38 B=155100 s/sec=2022.87 tok/sec=102048.00 u/sec=22.62 ppl=6.377889 [2023-12-17:13:13:52:INFO:sockeye.training:__call__] E=38 B=155150 s/sec=1968.50 tok/sec=103861.69 u/sec=22.53 ppl=6.381181 [2023-12-17:13:13:54:INFO:sockeye.training:__call__] E=38 B=155200 s/sec=1985.29 tok/sec=105872.77 u/sec=21.92 ppl=6.386023 [2023-12-17:13:13:56:INFO:sockeye.training:__call__] E=38 B=155250 s/sec=2100.15 tok/sec=98610.64 u/sec=23.19 ppl=6.390888 [2023-12-17:13:13:58:INFO:sockeye.training:__call__] E=38 B=155300 s/sec=1940.32 tok/sec=100679.87 u/sec=22.58 ppl=6.394376 [2023-12-17:13:14:01:INFO:sockeye.training:__call__] E=38 B=155350 s/sec=1999.72 tok/sec=102491.37 u/sec=22.89 ppl=6.399221 [2023-12-17:13:14:03:INFO:sockeye.training:__call__] E=38 B=155400 s/sec=2102.10 tok/sec=98498.33 u/sec=23.46 ppl=6.404183 [2023-12-17:13:14:05:INFO:sockeye.training:__call__] E=38 B=155450 s/sec=2102.54 tok/sec=97847.47 u/sec=23.09 ppl=6.409239 [2023-12-17:13:14:07:INFO:sockeye.training:__call__] E=38 B=155500 s/sec=2133.78 tok/sec=95298.87 u/sec=23.48 ppl=6.415632 [2023-12-17:13:14:09:INFO:sockeye.training:__call__] E=38 B=155550 s/sec=2194.52 tok/sec=98637.68 u/sec=23.37 ppl=6.420776 [2023-12-17:13:14:11:INFO:sockeye.training:__call__] E=38 B=155600 s/sec=2011.11 tok/sec=99358.40 u/sec=23.23 ppl=6.424887 [2023-12-17:13:14:13:INFO:sockeye.training:__call__] E=38 B=155650 s/sec=1989.66 tok/sec=99639.35 u/sec=23.24 ppl=6.428570 [2023-12-17:13:14:15:INFO:sockeye.training:__call__] E=38 B=155700 s/sec=2052.47 tok/sec=97339.29 u/sec=23.62 ppl=6.432443 [2023-12-17:13:14:18:INFO:sockeye.training:__call__] E=38 B=155750 s/sec=2163.62 tok/sec=96440.01 u/sec=23.36 ppl=6.437901 [2023-12-17:13:14:20:INFO:sockeye.training:__call__] E=38 B=155800 s/sec=1927.65 tok/sec=103448.33 u/sec=22.39 ppl=6.440087 [2023-12-17:13:14:22:INFO:sockeye.training:__call__] E=39 B=155850 s/sec=1935.09 tok/sec=95100.21 u/sec=22.48 ppl=6.433869 [2023-12-17:13:14:24:INFO:sockeye.training:__call__] E=39 B=155900 s/sec=2095.65 tok/sec=98748.10 u/sec=23.22 ppl=6.425990 [2023-12-17:13:14:26:INFO:sockeye.training:__call__] E=39 B=155950 s/sec=1806.27 tok/sec=104262.06 u/sec=22.71 ppl=6.417547 [2023-12-17:13:14:29:INFO:sockeye.training:__call__] E=39 B=156000 s/sec=1616.45 tok/sec=106734.52 u/sec=21.91 ppl=6.408377 [2023-12-17:13:14:29:INFO:sockeye.training:_create_checkpoint] Checkpoint [39] Updates=156000 Epoch=39 Samples=13901656 Time-cost=172.748 Updates/sec=23.155 [2023-12-17:13:14:29:INFO:sockeye.training:_create_checkpoint] Checkpoint [39] Train-ppl=6.408377 [2023-12-17:13:14:31:INFO:sockeye.training:_evaluate] Checkpoint [39] Validation-ppl=32.514162 Validation-bleu=0.171130 Validation-chrf=31.561320 Validation-rouge1=0.166971 Validation-rouge2=0.084283 Validation-rougel=0.160495 Validation-length-ratio-mse=1.275522 Validation-ter=95.130004 Validation-avg-sec-per-sent=0.003296 Validation-decode-walltime=1.647823 [2023-12-17:13:14:31:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 15 checkpoints, best so far: 32.780032 [2023-12-17:13:14:31:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:13:14:31:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [39] Learning-rate=0.000146 [2023-12-17:13:14:32:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:13:14:32:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=7, base_lr=0.0002, lr=0.00014580000000000002, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:13:14:32:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00039" [2023-12-17:13:14:34:INFO:sockeye.training:__call__] E=39 B=156050 s/sec=864.75 tok/sec=38822.00 u/sec=9.08 ppl=5.916778 [2023-12-17:13:14:36:INFO:sockeye.training:__call__] E=39 B=156100 s/sec=1968.52 tok/sec=101979.71 u/sec=23.04 ppl=5.904237 [2023-12-17:13:14:39:INFO:sockeye.training:__call__] E=39 B=156150 s/sec=2111.70 tok/sec=98555.96 u/sec=23.32 ppl=5.878092 [2023-12-17:13:14:41:INFO:sockeye.training:__call__] E=39 B=156200 s/sec=2149.66 tok/sec=97521.00 u/sec=23.28 ppl=5.897691 [2023-12-17:13:14:43:INFO:sockeye.training:__call__] E=39 B=156250 s/sec=1874.90 tok/sec=99695.19 u/sec=23.39 ppl=5.887016 [2023-12-17:13:14:45:INFO:sockeye.training:__call__] E=39 B=156300 s/sec=2074.53 tok/sec=99140.10 u/sec=22.79 ppl=5.888836 [2023-12-17:13:14:47:INFO:sockeye.training:__call__] E=39 B=156350 s/sec=2202.49 tok/sec=97314.65 u/sec=23.45 ppl=5.915540 [2023-12-17:13:14:49:INFO:sockeye.training:__call__] E=39 B=156400 s/sec=2276.11 tok/sec=96558.70 u/sec=23.67 ppl=5.936514 [2023-12-17:13:14:51:INFO:sockeye.training:__call__] E=39 B=156450 s/sec=1866.09 tok/sec=100598.84 u/sec=23.10 ppl=5.935355 [2023-12-17:13:14:54:INFO:sockeye.training:__call__] E=39 B=156500 s/sec=2269.97 tok/sec=97084.75 u/sec=23.72 ppl=5.955059 [2023-12-17:13:14:56:INFO:sockeye.training:__call__] E=39 B=156550 s/sec=2149.45 tok/sec=98590.27 u/sec=23.40 ppl=5.968644 [2023-12-17:13:14:58:INFO:sockeye.training:__call__] E=39 B=156600 s/sec=2045.39 tok/sec=100891.35 u/sec=22.79 ppl=5.978716 [2023-12-17:13:15:00:INFO:sockeye.training:__call__] E=39 B=156650 s/sec=2068.80 tok/sec=100751.71 u/sec=22.64 ppl=5.988388 [2023-12-17:13:15:02:INFO:sockeye.training:__call__] E=39 B=156700 s/sec=2063.64 tok/sec=100664.75 u/sec=22.87 ppl=5.990419 [2023-12-17:13:15:05:INFO:sockeye.training:__call__] E=39 B=156750 s/sec=1760.72 tok/sec=104722.11 u/sec=22.14 ppl=5.993462 [2023-12-17:13:15:07:INFO:sockeye.training:__call__] E=39 B=156800 s/sec=2155.58 tok/sec=98727.01 u/sec=23.15 ppl=6.000888 [2023-12-17:13:15:09:INFO:sockeye.training:__call__] E=39 B=156850 s/sec=2222.23 tok/sec=98428.01 u/sec=23.07 ppl=6.010321 [2023-12-17:13:15:11:INFO:sockeye.training:__call__] E=39 B=156900 s/sec=2163.79 tok/sec=96198.44 u/sec=23.48 ppl=6.023832 [2023-12-17:13:15:13:INFO:sockeye.training:__call__] E=39 B=156950 s/sec=2064.10 tok/sec=99343.88 u/sec=23.20 ppl=6.024859 [2023-12-17:13:15:15:INFO:sockeye.training:__call__] E=39 B=157000 s/sec=2068.53 tok/sec=96100.32 u/sec=23.51 ppl=6.030636 [2023-12-17:13:15:17:INFO:sockeye.training:__call__] E=39 B=157050 s/sec=1993.69 tok/sec=100765.07 u/sec=23.29 ppl=6.037887 [2023-12-17:13:15:20:INFO:sockeye.training:__call__] E=39 B=157100 s/sec=2246.77 tok/sec=95036.81 u/sec=23.88 ppl=6.046317 [2023-12-17:13:15:22:INFO:sockeye.training:__call__] E=39 B=157150 s/sec=2048.94 tok/sec=101236.57 u/sec=22.79 ppl=6.051774 [2023-12-17:13:15:24:INFO:sockeye.training:__call__] E=39 B=157200 s/sec=2157.87 tok/sec=97297.49 u/sec=23.70 ppl=6.061513 [2023-12-17:13:15:26:INFO:sockeye.training:__call__] E=39 B=157250 s/sec=2086.63 tok/sec=97928.65 u/sec=23.54 ppl=6.071769 [2023-12-17:13:15:28:INFO:sockeye.training:__call__] E=39 B=157300 s/sec=2191.14 tok/sec=98382.60 u/sec=23.17 ppl=6.078649 [2023-12-17:13:15:30:INFO:sockeye.training:__call__] E=39 B=157350 s/sec=2206.43 tok/sec=96723.86 u/sec=23.37 ppl=6.086738 [2023-12-17:13:15:32:INFO:sockeye.training:__call__] E=39 B=157400 s/sec=2247.04 tok/sec=98288.34 u/sec=23.29 ppl=6.094894 [2023-12-17:13:15:35:INFO:sockeye.training:__call__] E=39 B=157450 s/sec=1692.57 tok/sec=104353.93 u/sec=22.32 ppl=6.099536 [2023-12-17:13:15:37:INFO:sockeye.training:__call__] E=39 B=157500 s/sec=1813.96 tok/sec=101089.38 u/sec=22.63 ppl=6.102657 [2023-12-17:13:15:39:INFO:sockeye.training:__call__] E=39 B=157550 s/sec=1981.91 tok/sec=101062.70 u/sec=22.94 ppl=6.105844 [2023-12-17:13:15:41:INFO:sockeye.training:__call__] E=39 B=157600 s/sec=2418.55 tok/sec=93769.86 u/sec=23.69 ppl=6.118072 [2023-12-17:13:15:43:INFO:sockeye.training:__call__] E=39 B=157650 s/sec=2124.90 tok/sec=98642.94 u/sec=23.18 ppl=6.126708 [2023-12-17:13:15:45:INFO:sockeye.training:__call__] E=39 B=157700 s/sec=1916.91 tok/sec=98592.30 u/sec=23.26 ppl=6.131432 [2023-12-17:13:15:48:INFO:sockeye.training:__call__] E=39 B=157750 s/sec=1963.13 tok/sec=101793.14 u/sec=22.64 ppl=6.134373 [2023-12-17:13:15:50:INFO:sockeye.training:__call__] E=39 B=157800 s/sec=2103.97 tok/sec=98615.84 u/sec=23.19 ppl=6.143084 [2023-12-17:13:15:52:INFO:sockeye.training:__call__] E=39 B=157850 s/sec=2003.24 tok/sec=98861.41 u/sec=23.49 ppl=6.146677 [2023-12-17:13:15:54:INFO:sockeye.training:__call__] E=39 B=157900 s/sec=1996.60 tok/sec=98894.94 u/sec=23.19 ppl=6.150246 [2023-12-17:13:15:56:INFO:sockeye.training:__call__] E=39 B=157950 s/sec=2141.02 tok/sec=94633.19 u/sec=23.90 ppl=6.156279 [2023-12-17:13:15:58:INFO:sockeye.training:__call__] E=39 B=158000 s/sec=2200.02 tok/sec=96861.17 u/sec=23.63 ppl=6.162397 [2023-12-17:13:16:00:INFO:sockeye.training:__call__] E=39 B=158050 s/sec=1999.09 tok/sec=99726.19 u/sec=23.01 ppl=6.168284 [2023-12-17:13:16:03:INFO:sockeye.training:__call__] E=39 B=158100 s/sec=2169.63 tok/sec=99058.52 u/sec=23.50 ppl=6.174817 [2023-12-17:13:16:05:INFO:sockeye.training:__call__] E=39 B=158150 s/sec=2338.67 tok/sec=92895.22 u/sec=24.24 ppl=6.182860 [2023-12-17:13:16:07:INFO:sockeye.training:__call__] E=39 B=158200 s/sec=1760.59 tok/sec=102566.90 u/sec=22.83 ppl=6.184680 [2023-12-17:13:16:09:INFO:sockeye.training:__call__] E=39 B=158250 s/sec=2175.40 tok/sec=94710.18 u/sec=23.85 ppl=6.191183 [2023-12-17:13:16:11:INFO:sockeye.training:__call__] E=39 B=158300 s/sec=2157.39 tok/sec=97742.12 u/sec=23.29 ppl=6.197492 [2023-12-17:13:16:13:INFO:sockeye.training:__call__] E=39 B=158350 s/sec=2064.05 tok/sec=100948.36 u/sec=22.87 ppl=6.201496 [2023-12-17:13:16:15:INFO:sockeye.training:__call__] E=39 B=158400 s/sec=2270.90 tok/sec=95370.20 u/sec=23.93 ppl=6.210072 [2023-12-17:13:16:18:INFO:sockeye.training:__call__] E=39 B=158450 s/sec=1888.83 tok/sec=104933.29 u/sec=22.66 ppl=6.213493 [2023-12-17:13:16:20:INFO:sockeye.training:__call__] E=39 B=158500 s/sec=1975.52 tok/sec=102253.65 u/sec=22.41 ppl=6.218313 [2023-12-17:13:16:22:INFO:sockeye.training:__call__] E=39 B=158550 s/sec=1911.36 tok/sec=99524.80 u/sec=23.29 ppl=6.221132 [2023-12-17:13:16:24:INFO:sockeye.training:__call__] E=39 B=158600 s/sec=2172.72 tok/sec=106555.49 u/sec=22.12 ppl=6.226343 [2023-12-17:13:16:26:INFO:sockeye.training:__call__] E=39 B=158650 s/sec=2221.64 tok/sec=98008.59 u/sec=23.57 ppl=6.231390 [2023-12-17:13:16:29:INFO:sockeye.training:__call__] E=39 B=158700 s/sec=1882.90 tok/sec=104184.73 u/sec=22.33 ppl=6.235431 [2023-12-17:13:16:31:INFO:sockeye.training:__call__] E=39 B=158750 s/sec=1967.57 tok/sec=97489.04 u/sec=23.56 ppl=6.240676 [2023-12-17:13:16:33:INFO:sockeye.training:__call__] E=39 B=158800 s/sec=2008.44 tok/sec=98267.01 u/sec=23.46 ppl=6.245091 [2023-12-17:13:16:35:INFO:sockeye.training:__call__] E=39 B=158850 s/sec=2174.48 tok/sec=97205.10 u/sec=23.64 ppl=6.250997 [2023-12-17:13:16:37:INFO:sockeye.training:__call__] E=39 B=158900 s/sec=1955.77 tok/sec=101813.60 u/sec=22.22 ppl=6.254607 [2023-12-17:13:16:39:INFO:sockeye.training:__call__] E=39 B=158950 s/sec=2117.09 tok/sec=100161.88 u/sec=23.25 ppl=6.257361 [2023-12-17:13:16:42:INFO:sockeye.training:__call__] E=39 B=159000 s/sec=2039.82 tok/sec=98536.73 u/sec=23.26 ppl=6.260701 [2023-12-17:13:16:44:INFO:sockeye.training:__call__] E=39 B=159050 s/sec=2363.68 tok/sec=96375.18 u/sec=23.41 ppl=6.266421 [2023-12-17:13:16:46:INFO:sockeye.training:__call__] E=39 B=159100 s/sec=1834.66 tok/sec=101008.91 u/sec=22.93 ppl=6.267284 [2023-12-17:13:16:48:INFO:sockeye.training:__call__] E=39 B=159150 s/sec=1964.05 tok/sec=102404.55 u/sec=22.52 ppl=6.271066 [2023-12-17:13:16:50:INFO:sockeye.training:__call__] E=39 B=159200 s/sec=2334.38 tok/sec=93375.39 u/sec=24.24 ppl=6.277194 [2023-12-17:13:16:52:INFO:sockeye.training:__call__] E=39 B=159250 s/sec=2151.66 tok/sec=94154.14 u/sec=23.84 ppl=6.281585 [2023-12-17:13:16:54:INFO:sockeye.training:__call__] E=39 B=159300 s/sec=2230.20 tok/sec=94819.84 u/sec=23.83 ppl=6.287224 [2023-12-17:13:16:56:INFO:sockeye.training:__call__] E=39 B=159350 s/sec=2360.01 tok/sec=93479.53 u/sec=23.98 ppl=6.294335 [2023-12-17:13:16:59:INFO:sockeye.training:__call__] E=39 B=159400 s/sec=2014.94 tok/sec=100377.76 u/sec=23.06 ppl=6.298707 [2023-12-17:13:17:01:INFO:sockeye.training:__call__] E=39 B=159450 s/sec=1840.83 tok/sec=101888.52 u/sec=22.83 ppl=6.300949 [2023-12-17:13:17:03:INFO:sockeye.training:__call__] E=39 B=159500 s/sec=1851.51 tok/sec=103882.99 u/sec=22.13 ppl=6.303610 [2023-12-17:13:17:05:INFO:sockeye.training:__call__] E=39 B=159550 s/sec=2105.49 tok/sec=98266.15 u/sec=23.50 ppl=6.308840 [2023-12-17:13:17:07:INFO:sockeye.training:__call__] E=39 B=159600 s/sec=2240.96 tok/sec=95886.09 u/sec=23.58 ppl=6.314474 [2023-12-17:13:17:09:INFO:sockeye.training:__call__] E=39 B=159650 s/sec=1963.69 tok/sec=102301.97 u/sec=22.85 ppl=6.319452 [2023-12-17:13:17:12:INFO:sockeye.training:__call__] E=39 B=159700 s/sec=2197.19 tok/sec=96418.32 u/sec=23.72 ppl=6.324760 [2023-12-17:13:17:14:INFO:sockeye.training:__call__] E=39 B=159750 s/sec=2003.25 tok/sec=99432.09 u/sec=23.06 ppl=6.328986 [2023-12-17:13:17:16:INFO:sockeye.training:__call__] E=39 B=159800 s/sec=1907.37 tok/sec=99991.55 u/sec=22.97 ppl=6.333391 [2023-12-17:13:17:18:INFO:sockeye.training:__call__] E=40 B=159850 s/sec=1965.74 tok/sec=97988.98 u/sec=21.67 ppl=6.325783 [2023-12-17:13:17:20:INFO:sockeye.training:__call__] E=40 B=159900 s/sec=2200.04 tok/sec=98760.59 u/sec=23.54 ppl=6.318126 [2023-12-17:13:17:22:INFO:sockeye.training:__call__] E=40 B=159950 s/sec=2025.14 tok/sec=98649.88 u/sec=23.01 ppl=6.310831 [2023-12-17:13:17:25:INFO:sockeye.training:__call__] E=40 B=160000 s/sec=1914.37 tok/sec=100743.57 u/sec=23.37 ppl=6.302920 [2023-12-17:13:17:25:INFO:sockeye.training:_create_checkpoint] Checkpoint [40] Updates=160000 Epoch=40 Samples=14259304 Time-cost=172.559 Updates/sec=23.180 [2023-12-17:13:17:25:INFO:sockeye.training:_create_checkpoint] Checkpoint [40] Train-ppl=6.302920 [2023-12-17:13:17:27:INFO:sockeye.training:_evaluate] Checkpoint [40] Validation-ppl=32.755771 Validation-bleu=0.167581 Validation-chrf=31.524286 Validation-rouge1=0.170321 Validation-rouge2=0.081373 Validation-rougel=0.164012 Validation-length-ratio-mse=1.267890 Validation-ter=93.891870 Validation-avg-sec-per-sent=0.003265 Validation-decode-walltime=1.632673 [2023-12-17:13:17:27:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 16 checkpoints, best so far: 32.780032 [2023-12-17:13:17:27:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:13:17:27:INFO:sockeye.lr_scheduler:new_evaluation_result] 8 checkpoints since improvement or rate scaling, lowering learning rate: 1.46e-04 -> 1.31e-04 [2023-12-17:13:17:27:INFO:sockeye.training:_adjust_learning_rate] Loading model parameters and optimizer states from best checkpoint: 24 [2023-12-17:13:17:27:INFO:sockeye.model:load_parameters] Loaded params from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.best" to "cuda:0" [2023-12-17:13:17:28:INFO:sockeye.training:_load_optimizer_state] Loaded optimizer state from "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/optimizer_best.pkl" [2023-12-17:13:17:28:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [40] Learning-rate=0.000146 [2023-12-17:13:17:28:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:13:17:28:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=0, base_lr=0.0002, lr=0.00013122000000000003, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:13:17:28:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00040" [2023-12-17:13:17:30:INFO:sockeye.training:__call__] E=40 B=160050 s/sec=773.55 tok/sec=35344.69 u/sec=8.77 ppl=6.971254 [2023-12-17:13:17:32:INFO:sockeye.training:__call__] E=40 B=160100 s/sec=2207.82 tok/sec=94940.00 u/sec=23.75 ppl=6.981327 [2023-12-17:13:17:35:INFO:sockeye.training:__call__] E=40 B=160150 s/sec=1910.19 tok/sec=103466.22 u/sec=22.44 ppl=6.966730 [2023-12-17:13:17:37:INFO:sockeye.training:__call__] E=40 B=160200 s/sec=2174.92 tok/sec=96757.22 u/sec=23.68 ppl=6.941122 [2023-12-17:13:17:39:INFO:sockeye.training:__call__] E=40 B=160250 s/sec=2009.95 tok/sec=100278.43 u/sec=22.84 ppl=6.930059 [2023-12-17:13:17:41:INFO:sockeye.training:__call__] E=40 B=160300 s/sec=2148.34 tok/sec=97892.35 u/sec=23.47 ppl=6.934282 [2023-12-17:13:17:43:INFO:sockeye.training:__call__] E=40 B=160350 s/sec=1831.88 tok/sec=103978.14 u/sec=22.67 ppl=6.911379 [2023-12-17:13:17:45:INFO:sockeye.training:__call__] E=40 B=160400 s/sec=2165.03 tok/sec=99103.91 u/sec=23.09 ppl=6.913961 [2023-12-17:13:17:48:INFO:sockeye.training:__call__] E=40 B=160450 s/sec=1945.35 tok/sec=102783.75 u/sec=22.47 ppl=6.920765 [2023-12-17:13:17:50:INFO:sockeye.training:__call__] E=40 B=160500 s/sec=2228.75 tok/sec=95517.67 u/sec=23.69 ppl=6.942921 [2023-12-17:13:17:52:INFO:sockeye.training:__call__] E=40 B=160550 s/sec=1780.81 tok/sec=101354.62 u/sec=23.14 ppl=6.946847 [2023-12-17:13:17:54:INFO:sockeye.training:__call__] E=40 B=160600 s/sec=2086.25 tok/sec=99452.07 u/sec=23.37 ppl=6.950767 [2023-12-17:13:17:56:INFO:sockeye.training:__call__] E=40 B=160650 s/sec=1741.81 tok/sec=103205.55 u/sec=22.13 ppl=6.953201 [2023-12-17:13:17:59:INFO:sockeye.training:__call__] E=40 B=160700 s/sec=2014.42 tok/sec=100829.63 u/sec=22.60 ppl=6.954211 [2023-12-17:13:18:01:INFO:sockeye.training:__call__] E=40 B=160750 s/sec=2010.68 tok/sec=101138.25 u/sec=23.31 ppl=6.956891 [2023-12-17:13:18:03:INFO:sockeye.training:__call__] E=40 B=160800 s/sec=2190.23 tok/sec=97592.10 u/sec=23.28 ppl=6.966717 [2023-12-17:13:18:05:INFO:sockeye.training:__call__] E=40 B=160850 s/sec=2045.51 tok/sec=99973.01 u/sec=23.29 ppl=6.975635 [2023-12-17:13:18:07:INFO:sockeye.training:__call__] E=40 B=160900 s/sec=2274.69 tok/sec=97093.06 u/sec=23.27 ppl=6.988377 [2023-12-17:13:18:09:INFO:sockeye.training:__call__] E=40 B=160950 s/sec=2088.73 tok/sec=97374.73 u/sec=23.23 ppl=7.001802 [2023-12-17:13:18:11:INFO:sockeye.training:__call__] E=40 B=161000 s/sec=2165.77 tok/sec=95278.74 u/sec=23.71 ppl=7.010903 [2023-12-17:13:18:14:INFO:sockeye.training:__call__] E=40 B=161050 s/sec=2035.28 tok/sec=98140.59 u/sec=23.30 ppl=7.017215 [2023-12-17:13:18:16:INFO:sockeye.training:__call__] E=40 B=161100 s/sec=2217.71 tok/sec=97579.18 u/sec=23.41 ppl=7.023149 [2023-12-17:13:18:18:INFO:sockeye.training:__call__] E=40 B=161150 s/sec=2001.98 tok/sec=97228.79 u/sec=22.71 ppl=7.034241 [2023-12-17:13:18:20:INFO:sockeye.training:__call__] E=40 B=161200 s/sec=2152.71 tok/sec=98280.66 u/sec=23.48 ppl=7.036585 [2023-12-17:13:18:22:INFO:sockeye.training:__call__] E=40 B=161250 s/sec=2190.38 tok/sec=96301.09 u/sec=23.64 ppl=7.048106 [2023-12-17:13:18:24:INFO:sockeye.training:__call__] E=40 B=161300 s/sec=1984.82 tok/sec=100397.27 u/sec=23.02 ppl=7.052226 [2023-12-17:13:18:27:INFO:sockeye.training:__call__] E=40 B=161350 s/sec=1989.03 tok/sec=100090.39 u/sec=22.69 ppl=7.060607 [2023-12-17:13:18:29:INFO:sockeye.training:__call__] E=40 B=161400 s/sec=2021.31 tok/sec=100180.16 u/sec=23.05 ppl=7.065776 [2023-12-17:13:18:31:INFO:sockeye.training:__call__] E=40 B=161450 s/sec=2134.09 tok/sec=96895.77 u/sec=23.52 ppl=7.076031 [2023-12-17:13:18:33:INFO:sockeye.training:__call__] E=40 B=161500 s/sec=2105.61 tok/sec=95363.37 u/sec=23.71 ppl=7.081216 [2023-12-17:13:18:35:INFO:sockeye.training:__call__] E=40 B=161550 s/sec=2119.18 tok/sec=99797.31 u/sec=23.11 ppl=7.088498 [2023-12-17:13:18:37:INFO:sockeye.training:__call__] E=40 B=161600 s/sec=2143.20 tok/sec=97699.90 u/sec=23.50 ppl=7.098004 [2023-12-17:13:18:39:INFO:sockeye.training:__call__] E=40 B=161650 s/sec=2006.88 tok/sec=100146.11 u/sec=22.93 ppl=7.098984 [2023-12-17:13:18:42:INFO:sockeye.training:__call__] E=40 B=161700 s/sec=2158.36 tok/sec=97303.42 u/sec=23.10 ppl=7.107818 [2023-12-17:13:18:44:INFO:sockeye.training:__call__] E=40 B=161750 s/sec=2407.15 tok/sec=94234.89 u/sec=23.92 ppl=7.114989 [2023-12-17:13:18:46:INFO:sockeye.training:__call__] E=40 B=161800 s/sec=2055.75 tok/sec=97799.41 u/sec=23.62 ppl=7.121218 [2023-12-17:13:18:48:INFO:sockeye.training:__call__] E=40 B=161850 s/sec=2315.43 tok/sec=95168.91 u/sec=23.45 ppl=7.129881 [2023-12-17:13:18:50:INFO:sockeye.training:__call__] E=40 B=161900 s/sec=2191.11 tok/sec=97976.48 u/sec=23.33 ppl=7.132946 [2023-12-17:13:18:52:INFO:sockeye.training:__call__] E=40 B=161950 s/sec=1964.57 tok/sec=99418.93 u/sec=23.25 ppl=7.136670 [2023-12-17:13:18:54:INFO:sockeye.training:__call__] E=40 B=162000 s/sec=1921.52 tok/sec=100740.27 u/sec=22.92 ppl=7.141824 [2023-12-17:13:18:57:INFO:sockeye.training:__call__] E=40 B=162050 s/sec=1890.48 tok/sec=102882.46 u/sec=22.42 ppl=7.145756 [2023-12-17:13:18:59:INFO:sockeye.training:__call__] E=40 B=162100 s/sec=2196.95 tok/sec=96332.55 u/sec=23.67 ppl=7.151042 [2023-12-17:13:19:01:INFO:sockeye.training:__call__] E=40 B=162150 s/sec=2101.64 tok/sec=98489.67 u/sec=23.33 ppl=7.156716 [2023-12-17:13:19:03:INFO:sockeye.training:__call__] E=40 B=162200 s/sec=2369.22 tok/sec=94160.39 u/sec=23.77 ppl=7.166190 [2023-12-17:13:19:05:INFO:sockeye.training:__call__] E=40 B=162250 s/sec=2001.14 tok/sec=102727.37 u/sec=22.86 ppl=7.171398 [2023-12-17:13:19:07:INFO:sockeye.training:__call__] E=40 B=162300 s/sec=2149.98 tok/sec=97159.91 u/sec=23.13 ppl=7.176545 [2023-12-17:13:19:10:INFO:sockeye.training:__call__] E=40 B=162350 s/sec=1833.47 tok/sec=101891.42 u/sec=22.65 ppl=7.181568 [2023-12-17:13:19:12:INFO:sockeye.training:__call__] E=40 B=162400 s/sec=2063.31 tok/sec=100373.44 u/sec=22.66 ppl=7.187084 [2023-12-17:13:19:14:INFO:sockeye.training:__call__] E=40 B=162450 s/sec=2138.60 tok/sec=99099.55 u/sec=23.33 ppl=7.195262 [2023-12-17:13:19:16:INFO:sockeye.training:__call__] E=40 B=162500 s/sec=2220.74 tok/sec=96994.88 u/sec=23.37 ppl=7.203703 [2023-12-17:13:19:18:INFO:sockeye.training:__call__] E=40 B=162550 s/sec=1887.80 tok/sec=102790.17 u/sec=22.87 ppl=7.207822 [2023-12-17:13:19:20:INFO:sockeye.training:__call__] E=40 B=162600 s/sec=2013.97 tok/sec=99494.49 u/sec=23.22 ppl=7.211508 [2023-12-17:13:19:23:INFO:sockeye.training:__call__] E=40 B=162650 s/sec=1888.06 tok/sec=101799.87 u/sec=23.09 ppl=7.214044 [2023-12-17:13:19:25:INFO:sockeye.training:__call__] E=40 B=162700 s/sec=2219.67 tok/sec=95558.55 u/sec=23.51 ppl=7.223162 [2023-12-17:13:19:27:INFO:sockeye.training:__call__] E=40 B=162750 s/sec=2134.23 tok/sec=98493.56 u/sec=23.20 ppl=7.229979 [2023-12-17:13:19:29:INFO:sockeye.training:__call__] E=40 B=162800 s/sec=2019.89 tok/sec=98805.65 u/sec=22.95 ppl=7.236394 [2023-12-17:13:19:31:INFO:sockeye.training:__call__] E=40 B=162850 s/sec=2136.69 tok/sec=98699.44 u/sec=23.39 ppl=7.242859 [2023-12-17:13:19:33:INFO:sockeye.training:__call__] E=40 B=162900 s/sec=2061.95 tok/sec=97691.02 u/sec=23.30 ppl=7.251199 [2023-12-17:13:19:35:INFO:sockeye.training:__call__] E=40 B=162950 s/sec=2135.69 tok/sec=98010.53 u/sec=23.30 ppl=7.254845 [2023-12-17:13:19:38:INFO:sockeye.training:__call__] E=40 B=163000 s/sec=2311.24 tok/sec=94069.51 u/sec=23.88 ppl=7.261404 [2023-12-17:13:19:40:INFO:sockeye.training:__call__] E=40 B=163050 s/sec=1854.42 tok/sec=103553.85 u/sec=22.95 ppl=7.264887 [2023-12-17:13:19:42:INFO:sockeye.training:__call__] E=40 B=163100 s/sec=1914.87 tok/sec=100408.14 u/sec=22.88 ppl=7.269861 [2023-12-17:13:19:44:INFO:sockeye.training:__call__] E=40 B=163150 s/sec=2007.44 tok/sec=108920.87 u/sec=21.34 ppl=7.273861 [2023-12-17:13:19:46:INFO:sockeye.training:__call__] E=40 B=163200 s/sec=1820.51 tok/sec=103905.93 u/sec=22.49 ppl=7.275406 [2023-12-17:13:19:49:INFO:sockeye.training:__call__] E=40 B=163250 s/sec=2126.73 tok/sec=96492.11 u/sec=23.48 ppl=7.281839 [2023-12-17:13:19:51:INFO:sockeye.training:__call__] E=40 B=163300 s/sec=1951.22 tok/sec=98622.50 u/sec=23.36 ppl=7.283553 [2023-12-17:13:19:53:INFO:sockeye.training:__call__] E=40 B=163350 s/sec=2130.55 tok/sec=95803.24 u/sec=23.48 ppl=7.290226 [2023-12-17:13:19:55:INFO:sockeye.training:__call__] E=40 B=163400 s/sec=2243.24 tok/sec=94966.55 u/sec=23.93 ppl=7.296059 [2023-12-17:13:19:57:INFO:sockeye.training:__call__] E=40 B=163450 s/sec=2068.14 tok/sec=99091.23 u/sec=23.37 ppl=7.300733 [2023-12-17:13:19:59:INFO:sockeye.training:__call__] E=40 B=163500 s/sec=1951.46 tok/sec=102772.28 u/sec=22.50 ppl=7.304328 [2023-12-17:13:20:01:INFO:sockeye.training:__call__] E=40 B=163550 s/sec=2091.76 tok/sec=99980.12 u/sec=23.68 ppl=7.307709 [2023-12-17:13:20:04:INFO:sockeye.training:__call__] E=40 B=163600 s/sec=2041.26 tok/sec=99467.65 u/sec=23.24 ppl=7.309578 [2023-12-17:13:20:06:INFO:sockeye.training:__call__] E=40 B=163650 s/sec=2030.77 tok/sec=101583.28 u/sec=23.25 ppl=7.313850 [2023-12-17:13:20:08:INFO:sockeye.training:__call__] E=40 B=163700 s/sec=2052.27 tok/sec=101135.71 u/sec=23.32 ppl=7.317833 [2023-12-17:13:20:10:INFO:sockeye.training:__call__] E=40 B=163750 s/sec=2178.15 tok/sec=99840.03 u/sec=23.59 ppl=7.322287 [2023-12-17:13:20:12:INFO:sockeye.training:__call__] E=41 B=163800 s/sec=1730.00 tok/sec=97159.92 u/sec=21.89 ppl=7.324189 [2023-12-17:13:20:14:INFO:sockeye.training:__call__] E=41 B=163850 s/sec=2178.00 tok/sec=97289.27 u/sec=23.72 ppl=7.314700 [2023-12-17:13:20:17:INFO:sockeye.training:__call__] E=41 B=163900 s/sec=2053.15 tok/sec=99417.17 u/sec=23.33 ppl=7.304136 [2023-12-17:13:20:19:INFO:sockeye.training:__call__] E=41 B=163950 s/sec=2177.63 tok/sec=100341.54 u/sec=23.19 ppl=7.295217 [2023-12-17:13:20:21:INFO:sockeye.training:__call__] E=41 B=164000 s/sec=2408.59 tok/sec=94127.58 u/sec=24.05 ppl=7.287471 [2023-12-17:13:20:21:INFO:sockeye.training:_create_checkpoint] Checkpoint [41] Updates=164000 Epoch=41 Samples=14616912 Time-cost=172.533 Updates/sec=23.184 [2023-12-17:13:20:21:INFO:sockeye.training:_create_checkpoint] Checkpoint [41] Train-ppl=7.287471 [2023-12-17:13:20:23:INFO:sockeye.training:_evaluate] Checkpoint [41] Validation-ppl=28.929884 Validation-bleu=0.168571 Validation-chrf=31.142122 Validation-rouge1=0.164500 Validation-rouge2=0.079600 Validation-rougel=0.157009 Validation-length-ratio-mse=1.270861 Validation-ter=94.510937 Validation-avg-sec-per-sent=0.003281 Validation-decode-walltime=1.640489 [2023-12-17:13:20:23:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 17 checkpoints, best so far: 32.780032 [2023-12-17:13:20:23:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:13:20:23:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [41] Learning-rate=0.000131 [2023-12-17:13:20:24:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:13:20:24:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=1, base_lr=0.0002, lr=0.00013122000000000003, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:13:20:24:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00041" [2023-12-17:13:20:26:INFO:sockeye.training:__call__] E=41 B=164050 s/sec=786.43 tok/sec=37815.47 u/sec=8.73 ppl=6.683020 [2023-12-17:13:20:29:INFO:sockeye.training:__call__] E=41 B=164100 s/sec=2115.07 tok/sec=99613.97 u/sec=22.95 ppl=6.706148 [2023-12-17:13:20:31:INFO:sockeye.training:__call__] E=41 B=164150 s/sec=2103.79 tok/sec=101857.03 u/sec=22.79 ppl=6.657124 [2023-12-17:13:20:33:INFO:sockeye.training:__call__] E=41 B=164200 s/sec=2034.85 tok/sec=100147.96 u/sec=22.75 ppl=6.654903 [2023-12-17:13:20:35:INFO:sockeye.training:__call__] E=41 B=164250 s/sec=1899.13 tok/sec=100040.97 u/sec=23.37 ppl=6.656226 [2023-12-17:13:20:37:INFO:sockeye.training:__call__] E=41 B=164300 s/sec=2207.23 tok/sec=97923.50 u/sec=23.30 ppl=6.679209 [2023-12-17:13:20:40:INFO:sockeye.training:__call__] E=41 B=164350 s/sec=1877.89 tok/sec=103524.76 u/sec=22.44 ppl=6.672182 [2023-12-17:13:20:42:INFO:sockeye.training:__call__] E=41 B=164400 s/sec=2031.13 tok/sec=101665.35 u/sec=22.63 ppl=6.677978 [2023-12-17:13:20:44:INFO:sockeye.training:__call__] E=41 B=164450 s/sec=2023.62 tok/sec=95843.74 u/sec=24.00 ppl=6.675015 [2023-12-17:13:20:46:INFO:sockeye.training:__call__] E=41 B=164500 s/sec=2006.88 tok/sec=98533.22 u/sec=23.58 ppl=6.683593 [2023-12-17:13:20:48:INFO:sockeye.training:__call__] E=41 B=164550 s/sec=2259.67 tok/sec=97874.29 u/sec=23.31 ppl=6.698227 [2023-12-17:13:20:50:INFO:sockeye.training:__call__] E=41 B=164600 s/sec=1984.33 tok/sec=100824.46 u/sec=23.05 ppl=6.703074 [2023-12-17:13:20:52:INFO:sockeye.training:__call__] E=41 B=164650 s/sec=2075.48 tok/sec=96526.72 u/sec=23.71 ppl=6.706315 [2023-12-17:13:20:55:INFO:sockeye.training:__call__] E=41 B=164700 s/sec=1798.16 tok/sec=101197.25 u/sec=22.98 ppl=6.710906 [2023-12-17:13:20:57:INFO:sockeye.training:__call__] E=41 B=164750 s/sec=2026.42 tok/sec=97146.42 u/sec=23.76 ppl=6.718350 [2023-12-17:13:20:59:INFO:sockeye.training:__call__] E=41 B=164800 s/sec=2131.37 tok/sec=98937.51 u/sec=23.49 ppl=6.730959 [2023-12-17:13:21:01:INFO:sockeye.training:__call__] E=41 B=164850 s/sec=2069.38 tok/sec=102044.47 u/sec=23.30 ppl=6.736213 [2023-12-17:13:21:03:INFO:sockeye.training:__call__] E=41 B=164900 s/sec=2032.46 tok/sec=99208.03 u/sec=23.44 ppl=6.744864 [2023-12-17:13:21:05:INFO:sockeye.training:__call__] E=41 B=164950 s/sec=1820.59 tok/sec=107428.82 u/sec=22.05 ppl=6.742694 [2023-12-17:13:21:07:INFO:sockeye.training:__call__] E=41 B=165000 s/sec=2309.83 tok/sec=97339.15 u/sec=23.71 ppl=6.755218 [2023-12-17:13:21:10:INFO:sockeye.training:__call__] E=41 B=165050 s/sec=2057.04 tok/sec=101510.66 u/sec=23.29 ppl=6.767109 [2023-12-17:13:21:12:INFO:sockeye.training:__call__] E=41 B=165100 s/sec=2017.93 tok/sec=102877.69 u/sec=22.93 ppl=6.775086 [2023-12-17:13:21:14:INFO:sockeye.training:__call__] E=41 B=165150 s/sec=1833.64 tok/sec=101873.80 u/sec=23.44 ppl=6.780168 [2023-12-17:13:21:16:INFO:sockeye.training:__call__] E=41 B=165200 s/sec=2053.37 tok/sec=101290.17 u/sec=22.20 ppl=6.789635 [2023-12-17:13:21:18:INFO:sockeye.training:__call__] E=41 B=165250 s/sec=2080.60 tok/sec=98940.91 u/sec=23.39 ppl=6.797189 [2023-12-17:13:21:20:INFO:sockeye.training:__call__] E=41 B=165300 s/sec=2095.23 tok/sec=95869.34 u/sec=23.85 ppl=6.810973 [2023-12-17:13:21:23:INFO:sockeye.training:__call__] E=41 B=165350 s/sec=1894.93 tok/sec=102326.26 u/sec=23.13 ppl=6.812566 [2023-12-17:13:21:25:INFO:sockeye.training:__call__] E=41 B=165400 s/sec=2083.10 tok/sec=98913.46 u/sec=23.33 ppl=6.822780 [2023-12-17:13:21:27:INFO:sockeye.training:__call__] E=41 B=165450 s/sec=2034.79 tok/sec=100319.10 u/sec=23.00 ppl=6.825690 [2023-12-17:13:21:29:INFO:sockeye.training:__call__] E=41 B=165500 s/sec=1864.70 tok/sec=99671.08 u/sec=23.50 ppl=6.828389 [2023-12-17:13:21:31:INFO:sockeye.training:__call__] E=41 B=165550 s/sec=2229.78 tok/sec=99044.27 u/sec=23.54 ppl=6.838280 [2023-12-17:13:21:33:INFO:sockeye.training:__call__] E=41 B=165600 s/sec=2141.26 tok/sec=99306.70 u/sec=23.40 ppl=6.844398 [2023-12-17:13:21:35:INFO:sockeye.training:__call__] E=41 B=165650 s/sec=2000.08 tok/sec=101347.16 u/sec=23.19 ppl=6.849517 [2023-12-17:13:21:38:INFO:sockeye.training:__call__] E=41 B=165700 s/sec=2155.90 tok/sec=97653.48 u/sec=23.60 ppl=6.858972 [2023-12-17:13:21:40:INFO:sockeye.training:__call__] E=41 B=165750 s/sec=2224.76 tok/sec=97312.31 u/sec=23.73 ppl=6.865616 [2023-12-17:13:21:42:INFO:sockeye.training:__call__] E=41 B=165800 s/sec=1949.51 tok/sec=101890.56 u/sec=23.03 ppl=6.869193 [2023-12-17:13:21:44:INFO:sockeye.training:__call__] E=41 B=165850 s/sec=2159.96 tok/sec=97828.07 u/sec=23.44 ppl=6.878470 [2023-12-17:13:21:46:INFO:sockeye.training:__call__] E=41 B=165900 s/sec=2056.32 tok/sec=97673.41 u/sec=23.67 ppl=6.884696 [2023-12-17:13:21:48:INFO:sockeye.training:__call__] E=41 B=165950 s/sec=1885.86 tok/sec=101684.07 u/sec=22.71 ppl=6.888721 [2023-12-17:13:21:50:INFO:sockeye.training:__call__] E=41 B=166000 s/sec=2212.85 tok/sec=97111.73 u/sec=23.32 ppl=6.893977 [2023-12-17:13:21:53:INFO:sockeye.training:__call__] E=41 B=166050 s/sec=2213.66 tok/sec=96976.65 u/sec=23.69 ppl=6.901399 [2023-12-17:13:21:55:INFO:sockeye.training:__call__] E=41 B=166100 s/sec=2304.90 tok/sec=96205.93 u/sec=23.73 ppl=6.906341 [2023-12-17:13:21:57:INFO:sockeye.training:__call__] E=41 B=166150 s/sec=2034.99 tok/sec=99481.74 u/sec=23.47 ppl=6.913572 [2023-12-17:13:21:59:INFO:sockeye.training:__call__] E=41 B=166200 s/sec=2148.66 tok/sec=97555.32 u/sec=23.31 ppl=6.919218 [2023-12-17:13:22:01:INFO:sockeye.training:__call__] E=41 B=166250 s/sec=1961.93 tok/sec=101743.61 u/sec=22.75 ppl=6.925808 [2023-12-17:13:22:03:INFO:sockeye.training:__call__] E=41 B=166300 s/sec=2146.13 tok/sec=99412.25 u/sec=23.45 ppl=6.935134 [2023-12-17:13:22:06:INFO:sockeye.training:__call__] E=41 B=166350 s/sec=1760.60 tok/sec=104474.26 u/sec=22.41 ppl=6.939619 [2023-12-17:13:22:08:INFO:sockeye.training:__call__] E=41 B=166400 s/sec=2079.41 tok/sec=98909.19 u/sec=23.50 ppl=6.947509 [2023-12-17:13:22:10:INFO:sockeye.training:__call__] E=41 B=166450 s/sec=2038.09 tok/sec=99870.03 u/sec=23.46 ppl=6.953784 [2023-12-17:13:22:12:INFO:sockeye.training:__call__] E=41 B=166500 s/sec=2276.59 tok/sec=95891.80 u/sec=23.87 ppl=6.960770 [2023-12-17:13:22:14:INFO:sockeye.training:__call__] E=41 B=166550 s/sec=1919.47 tok/sec=103759.82 u/sec=22.59 ppl=6.963835 [2023-12-17:13:22:16:INFO:sockeye.training:__call__] E=41 B=166600 s/sec=2205.39 tok/sec=98565.21 u/sec=23.64 ppl=6.970113 [2023-12-17:13:22:18:INFO:sockeye.training:__call__] E=41 B=166650 s/sec=2210.61 tok/sec=96916.62 u/sec=23.78 ppl=6.976808 [2023-12-17:13:22:20:INFO:sockeye.training:__call__] E=41 B=166700 s/sec=1945.71 tok/sec=102351.89 u/sec=22.94 ppl=6.982677 [2023-12-17:13:22:23:INFO:sockeye.training:__call__] E=41 B=166750 s/sec=2173.45 tok/sec=97419.22 u/sec=23.67 ppl=6.990032 [2023-12-17:13:22:25:INFO:sockeye.training:__call__] E=41 B=166800 s/sec=2268.93 tok/sec=95983.07 u/sec=24.16 ppl=6.997515 [2023-12-17:13:22:27:INFO:sockeye.training:__call__] E=41 B=166850 s/sec=2269.74 tok/sec=97865.88 u/sec=23.53 ppl=7.003375 [2023-12-17:13:22:29:INFO:sockeye.training:__call__] E=41 B=166900 s/sec=2216.48 tok/sec=95165.84 u/sec=24.09 ppl=7.010440 [2023-12-17:13:22:31:INFO:sockeye.training:__call__] E=41 B=166950 s/sec=1887.08 tok/sec=102249.41 u/sec=23.08 ppl=7.015408 [2023-12-17:13:22:33:INFO:sockeye.training:__call__] E=41 B=167000 s/sec=1966.47 tok/sec=102184.71 u/sec=22.47 ppl=7.019851 [2023-12-17:13:22:35:INFO:sockeye.training:__call__] E=41 B=167050 s/sec=1923.05 tok/sec=103837.02 u/sec=23.16 ppl=7.024492 [2023-12-17:13:22:38:INFO:sockeye.training:__call__] E=41 B=167100 s/sec=1979.90 tok/sec=100058.74 u/sec=23.09 ppl=7.028457 [2023-12-17:13:22:40:INFO:sockeye.training:__call__] E=41 B=167150 s/sec=2215.04 tok/sec=97165.21 u/sec=23.15 ppl=7.034305 [2023-12-17:13:22:42:INFO:sockeye.training:__call__] E=41 B=167200 s/sec=2094.36 tok/sec=107068.38 u/sec=21.93 ppl=7.039100 [2023-12-17:13:22:44:INFO:sockeye.training:__call__] E=41 B=167250 s/sec=2091.69 tok/sec=98926.63 u/sec=23.51 ppl=7.043964 [2023-12-17:13:22:46:INFO:sockeye.training:__call__] E=41 B=167300 s/sec=1993.52 tok/sec=100207.52 u/sec=23.07 ppl=7.048765 [2023-12-17:13:22:48:INFO:sockeye.training:__call__] E=41 B=167350 s/sec=2047.06 tok/sec=98198.41 u/sec=23.61 ppl=7.054390 [2023-12-17:13:22:51:INFO:sockeye.training:__call__] E=41 B=167400 s/sec=2215.82 tok/sec=98187.27 u/sec=23.47 ppl=7.061796 [2023-12-17:13:22:53:INFO:sockeye.training:__call__] E=41 B=167450 s/sec=2158.60 tok/sec=97995.94 u/sec=23.34 ppl=7.068126 [2023-12-17:13:22:55:INFO:sockeye.training:__call__] E=41 B=167500 s/sec=1980.12 tok/sec=101734.06 u/sec=22.92 ppl=7.073173 [2023-12-17:13:22:57:INFO:sockeye.training:__call__] E=41 B=167550 s/sec=1866.40 tok/sec=102891.42 u/sec=22.69 ppl=7.076700 [2023-12-17:13:22:59:INFO:sockeye.training:__call__] E=41 B=167600 s/sec=1964.75 tok/sec=102357.97 u/sec=22.95 ppl=7.080580 [2023-12-17:13:23:01:INFO:sockeye.training:__call__] E=41 B=167650 s/sec=2089.63 tok/sec=100420.60 u/sec=23.07 ppl=7.085158 [2023-12-17:13:23:04:INFO:sockeye.training:__call__] E=41 B=167700 s/sec=2203.77 tok/sec=95097.97 u/sec=24.12 ppl=7.089622 [2023-12-17:13:23:06:INFO:sockeye.training:__call__] E=41 B=167750 s/sec=2340.24 tok/sec=92960.66 u/sec=24.14 ppl=7.096716 [2023-12-17:13:23:08:INFO:sockeye.training:__call__] E=42 B=167800 s/sec=1911.48 tok/sec=95953.81 u/sec=21.18 ppl=7.098501 [2023-12-17:13:23:10:INFO:sockeye.training:__call__] E=42 B=167850 s/sec=2044.15 tok/sec=100621.32 u/sec=23.27 ppl=7.089217 [2023-12-17:13:23:12:INFO:sockeye.training:__call__] E=42 B=167900 s/sec=2290.87 tok/sec=94119.81 u/sec=24.27 ppl=7.079921 [2023-12-17:13:23:14:INFO:sockeye.training:__call__] E=42 B=167950 s/sec=1943.10 tok/sec=103195.86 u/sec=23.22 ppl=7.069359 [2023-12-17:13:23:16:INFO:sockeye.training:__call__] E=42 B=168000 s/sec=2246.89 tok/sec=97720.98 u/sec=23.48 ppl=7.061089 [2023-12-17:13:23:16:INFO:sockeye.training:_create_checkpoint] Checkpoint [42] Updates=168000 Epoch=42 Samples=14972672 Time-cost=172.099 Updates/sec=23.242 [2023-12-17:13:23:16:INFO:sockeye.training:_create_checkpoint] Checkpoint [42] Train-ppl=7.061089 [2023-12-17:13:23:19:INFO:sockeye.training:_evaluate] Checkpoint [42] Validation-ppl=30.015921 Validation-bleu=0.153314 Validation-chrf=29.776083 Validation-rouge1=0.152118 Validation-rouge2=0.074443 Validation-rougel=0.145181 Validation-length-ratio-mse=1.337104 Validation-ter=100.701610 Validation-avg-sec-per-sent=0.003569 Validation-decode-walltime=1.784608 [2023-12-17:13:23:19:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 18 checkpoints, best so far: 32.780032 [2023-12-17:13:23:19:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:13:23:19:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [42] Learning-rate=0.000131 [2023-12-17:13:23:20:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:13:23:20:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=2, base_lr=0.0002, lr=0.00013122000000000003, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:13:23:20:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00042" [2023-12-17:13:23:22:INFO:sockeye.training:__call__] E=42 B=168050 s/sec=668.74 tok/sec=39315.51 u/sec=8.43 ppl=6.357637 [2023-12-17:13:23:24:INFO:sockeye.training:__call__] E=42 B=168100 s/sec=2231.16 tok/sec=97332.51 u/sec=23.40 ppl=6.432803 [2023-12-17:13:23:27:INFO:sockeye.training:__call__] E=42 B=168150 s/sec=1985.15 tok/sec=101880.04 u/sec=22.89 ppl=6.427351 [2023-12-17:13:23:29:INFO:sockeye.training:__call__] E=42 B=168200 s/sec=2039.51 tok/sec=98166.32 u/sec=23.43 ppl=6.422682 [2023-12-17:13:23:31:INFO:sockeye.training:__call__] E=42 B=168250 s/sec=2068.92 tok/sec=98610.92 u/sec=23.68 ppl=6.433397 [2023-12-17:13:23:33:INFO:sockeye.training:__call__] E=42 B=168300 s/sec=1918.01 tok/sec=100785.43 u/sec=23.41 ppl=6.443893 [2023-12-17:13:23:35:INFO:sockeye.training:__call__] E=42 B=168350 s/sec=2106.67 tok/sec=98311.43 u/sec=23.60 ppl=6.443689 [2023-12-17:13:23:37:INFO:sockeye.training:__call__] E=42 B=168400 s/sec=2100.42 tok/sec=99335.87 u/sec=23.19 ppl=6.453297 [2023-12-17:13:23:40:INFO:sockeye.training:__call__] E=42 B=168450 s/sec=1948.79 tok/sec=100760.00 u/sec=23.11 ppl=6.450913 [2023-12-17:13:23:42:INFO:sockeye.training:__call__] E=42 B=168500 s/sec=1918.69 tok/sec=101894.71 u/sec=23.19 ppl=6.453696 [2023-12-17:13:23:44:INFO:sockeye.training:__call__] E=42 B=168550 s/sec=2218.24 tok/sec=96412.43 u/sec=23.54 ppl=6.470175 [2023-12-17:13:23:46:INFO:sockeye.training:__call__] E=42 B=168600 s/sec=1954.43 tok/sec=103759.76 u/sec=22.79 ppl=6.482918 [2023-12-17:13:23:48:INFO:sockeye.training:__call__] E=42 B=168650 s/sec=2125.22 tok/sec=99727.62 u/sec=23.47 ppl=6.493516 [2023-12-17:13:23:50:INFO:sockeye.training:__call__] E=42 B=168700 s/sec=2164.79 tok/sec=98599.59 u/sec=23.57 ppl=6.511535 [2023-12-17:13:23:52:INFO:sockeye.training:__call__] E=42 B=168750 s/sec=1979.48 tok/sec=99581.74 u/sec=23.17 ppl=6.518231 [2023-12-17:13:23:55:INFO:sockeye.training:__call__] E=42 B=168800 s/sec=1958.58 tok/sec=99440.35 u/sec=22.92 ppl=6.519952 [2023-12-17:13:23:57:INFO:sockeye.training:__call__] E=42 B=168850 s/sec=2121.84 tok/sec=97634.53 u/sec=23.35 ppl=6.532736 [2023-12-17:13:23:59:INFO:sockeye.training:__call__] E=42 B=168900 s/sec=2133.03 tok/sec=97211.71 u/sec=23.64 ppl=6.550252 [2023-12-17:13:24:01:INFO:sockeye.training:__call__] E=42 B=168950 s/sec=2152.54 tok/sec=99829.47 u/sec=23.52 ppl=6.558385 [2023-12-17:13:24:03:INFO:sockeye.training:__call__] E=42 B=169000 s/sec=2274.08 tok/sec=97714.33 u/sec=23.34 ppl=6.566204 [2023-12-17:13:24:05:INFO:sockeye.training:__call__] E=42 B=169050 s/sec=2050.44 tok/sec=101194.08 u/sec=22.80 ppl=6.577105 [2023-12-17:13:24:07:INFO:sockeye.training:__call__] E=42 B=169100 s/sec=2272.35 tok/sec=93754.83 u/sec=24.03 ppl=6.592352 [2023-12-17:13:24:10:INFO:sockeye.training:__call__] E=42 B=169150 s/sec=1977.66 tok/sec=102564.29 u/sec=22.51 ppl=6.596592 [2023-12-17:13:24:12:INFO:sockeye.training:__call__] E=42 B=169200 s/sec=1851.64 tok/sec=101745.56 u/sec=22.69 ppl=6.600904 [2023-12-17:13:24:14:INFO:sockeye.training:__call__] E=42 B=169250 s/sec=2054.33 tok/sec=98847.13 u/sec=23.39 ppl=6.608858 [2023-12-17:13:24:16:INFO:sockeye.training:__call__] E=42 B=169300 s/sec=2196.44 tok/sec=96049.66 u/sec=23.79 ppl=6.616402 [2023-12-17:13:24:18:INFO:sockeye.training:__call__] E=42 B=169350 s/sec=2030.36 tok/sec=100989.76 u/sec=22.62 ppl=6.623652 [2023-12-17:13:24:20:INFO:sockeye.training:__call__] E=42 B=169400 s/sec=1998.23 tok/sec=101184.12 u/sec=23.26 ppl=6.627983 [2023-12-17:13:24:23:INFO:sockeye.training:__call__] E=42 B=169450 s/sec=2028.30 tok/sec=99580.46 u/sec=22.84 ppl=6.637586 [2023-12-17:13:24:25:INFO:sockeye.training:__call__] E=42 B=169500 s/sec=2122.04 tok/sec=98738.58 u/sec=23.43 ppl=6.644237 [2023-12-17:13:24:27:INFO:sockeye.training:__call__] E=42 B=169550 s/sec=2249.70 tok/sec=102054.73 u/sec=22.83 ppl=6.654331 [2023-12-17:13:24:29:INFO:sockeye.training:__call__] E=42 B=169600 s/sec=2134.06 tok/sec=97328.23 u/sec=23.61 ppl=6.661881 [2023-12-17:13:24:31:INFO:sockeye.training:__call__] E=42 B=169650 s/sec=2268.63 tok/sec=95359.72 u/sec=24.20 ppl=6.670004 [2023-12-17:13:24:33:INFO:sockeye.training:__call__] E=42 B=169700 s/sec=2169.06 tok/sec=97887.39 u/sec=23.62 ppl=6.676400 [2023-12-17:13:24:35:INFO:sockeye.training:__call__] E=42 B=169750 s/sec=2202.53 tok/sec=99867.70 u/sec=23.21 ppl=6.684046 [2023-12-17:13:24:38:INFO:sockeye.training:__call__] E=42 B=169800 s/sec=2056.39 tok/sec=100060.07 u/sec=23.50 ppl=6.687210 [2023-12-17:13:24:40:INFO:sockeye.training:__call__] E=42 B=169850 s/sec=2274.21 tok/sec=96861.70 u/sec=23.61 ppl=6.697126 [2023-12-17:13:24:42:INFO:sockeye.training:__call__] E=42 B=169900 s/sec=1825.93 tok/sec=103065.47 u/sec=22.69 ppl=6.700372 [2023-12-17:13:24:44:INFO:sockeye.training:__call__] E=42 B=169950 s/sec=1840.98 tok/sec=104084.53 u/sec=22.74 ppl=6.702272 [2023-12-17:13:24:46:INFO:sockeye.training:__call__] E=42 B=170000 s/sec=1961.92 tok/sec=100657.84 u/sec=23.14 ppl=6.704605 [2023-12-17:13:24:48:INFO:sockeye.training:__call__] E=42 B=170050 s/sec=2368.96 tok/sec=94850.38 u/sec=23.92 ppl=6.713332 [2023-12-17:13:24:50:INFO:sockeye.training:__call__] E=42 B=170100 s/sec=2064.33 tok/sec=100538.12 u/sec=23.12 ppl=6.720062 [2023-12-17:13:24:53:INFO:sockeye.training:__call__] E=42 B=170150 s/sec=2225.22 tok/sec=96405.81 u/sec=23.98 ppl=6.730427 [2023-12-17:13:24:55:INFO:sockeye.training:__call__] E=42 B=170200 s/sec=2057.48 tok/sec=100446.47 u/sec=23.13 ppl=6.735803 [2023-12-17:13:24:57:INFO:sockeye.training:__call__] E=42 B=170250 s/sec=2083.32 tok/sec=96818.73 u/sec=23.89 ppl=6.741760 [2023-12-17:13:24:59:INFO:sockeye.training:__call__] E=42 B=170300 s/sec=2313.17 tok/sec=97767.45 u/sec=23.13 ppl=6.749636 [2023-12-17:13:25:01:INFO:sockeye.training:__call__] E=42 B=170350 s/sec=2027.32 tok/sec=97819.26 u/sec=23.33 ppl=6.755566 [2023-12-17:13:25:03:INFO:sockeye.training:__call__] E=42 B=170400 s/sec=2258.81 tok/sec=95502.24 u/sec=23.81 ppl=6.763564 [2023-12-17:13:25:05:INFO:sockeye.training:__call__] E=42 B=170450 s/sec=1963.71 tok/sec=99968.03 u/sec=23.60 ppl=6.768162 [2023-12-17:13:25:07:INFO:sockeye.training:__call__] E=42 B=170500 s/sec=2104.58 tok/sec=94748.50 u/sec=24.14 ppl=6.772804 [2023-12-17:13:25:10:INFO:sockeye.training:__call__] E=42 B=170550 s/sec=2202.74 tok/sec=99055.65 u/sec=23.49 ppl=6.777495 [2023-12-17:13:25:12:INFO:sockeye.training:__call__] E=42 B=170600 s/sec=2019.18 tok/sec=102757.08 u/sec=23.03 ppl=6.784047 [2023-12-17:13:25:14:INFO:sockeye.training:__call__] E=42 B=170650 s/sec=2122.31 tok/sec=95541.98 u/sec=23.77 ppl=6.787746 [2023-12-17:13:25:16:INFO:sockeye.training:__call__] E=42 B=170700 s/sec=1946.90 tok/sec=102546.31 u/sec=22.70 ppl=6.794873 [2023-12-17:13:25:18:INFO:sockeye.training:__call__] E=42 B=170750 s/sec=2134.74 tok/sec=99177.52 u/sec=23.37 ppl=6.800429 [2023-12-17:13:25:20:INFO:sockeye.training:__call__] E=42 B=170800 s/sec=1959.75 tok/sec=102252.86 u/sec=22.52 ppl=6.804887 [2023-12-17:13:25:23:INFO:sockeye.training:__call__] E=42 B=170850 s/sec=1874.86 tok/sec=103501.05 u/sec=22.41 ppl=6.807784 [2023-12-17:13:25:25:INFO:sockeye.training:__call__] E=42 B=170900 s/sec=1991.67 tok/sec=102729.59 u/sec=22.55 ppl=6.813122 [2023-12-17:13:25:27:INFO:sockeye.training:__call__] E=42 B=170950 s/sec=2253.32 tok/sec=96731.34 u/sec=23.43 ppl=6.818501 [2023-12-17:13:25:29:INFO:sockeye.training:__call__] E=42 B=171000 s/sec=2053.94 tok/sec=98014.12 u/sec=23.64 ppl=6.823223 [2023-12-17:13:25:31:INFO:sockeye.training:__call__] E=42 B=171050 s/sec=2065.99 tok/sec=100699.55 u/sec=23.02 ppl=6.828761 [2023-12-17:13:25:33:INFO:sockeye.training:__call__] E=42 B=171100 s/sec=1716.31 tok/sec=107212.47 u/sec=22.35 ppl=6.829751 [2023-12-17:13:25:36:INFO:sockeye.training:__call__] E=42 B=171150 s/sec=1849.60 tok/sec=102922.47 u/sec=23.26 ppl=6.833155 [2023-12-17:13:25:38:INFO:sockeye.training:__call__] E=42 B=171200 s/sec=1909.70 tok/sec=102867.38 u/sec=22.91 ppl=6.835193 [2023-12-17:13:25:40:INFO:sockeye.training:__call__] E=42 B=171250 s/sec=2030.42 tok/sec=100416.15 u/sec=23.33 ppl=6.839273 [2023-12-17:13:25:42:INFO:sockeye.training:__call__] E=42 B=171300 s/sec=2021.82 tok/sec=99374.83 u/sec=23.31 ppl=6.844536 [2023-12-17:13:25:44:INFO:sockeye.training:__call__] E=42 B=171350 s/sec=2224.61 tok/sec=97174.99 u/sec=23.53 ppl=6.849273 [2023-12-17:13:25:46:INFO:sockeye.training:__call__] E=42 B=171400 s/sec=1942.08 tok/sec=100585.59 u/sec=23.30 ppl=6.854977 [2023-12-17:13:25:48:INFO:sockeye.training:__call__] E=42 B=171450 s/sec=2207.85 tok/sec=97450.01 u/sec=23.79 ppl=6.857870 [2023-12-17:13:25:51:INFO:sockeye.training:__call__] E=42 B=171500 s/sec=2062.24 tok/sec=100136.71 u/sec=23.02 ppl=6.863324 [2023-12-17:13:25:53:INFO:sockeye.training:__call__] E=42 B=171550 s/sec=1952.99 tok/sec=103551.35 u/sec=22.27 ppl=6.867955 [2023-12-17:13:25:55:INFO:sockeye.training:__call__] E=42 B=171600 s/sec=1954.46 tok/sec=100693.80 u/sec=23.27 ppl=6.873399 [2023-12-17:13:25:57:INFO:sockeye.training:__call__] E=42 B=171650 s/sec=2006.37 tok/sec=101068.21 u/sec=22.97 ppl=6.878260 [2023-12-17:13:25:59:INFO:sockeye.training:__call__] E=42 B=171700 s/sec=2192.57 tok/sec=100066.90 u/sec=23.11 ppl=6.882217 [2023-12-17:13:26:01:INFO:sockeye.training:__call__] E=42 B=171750 s/sec=2298.69 tok/sec=96453.84 u/sec=23.79 ppl=6.887677 [2023-12-17:13:26:04:INFO:sockeye.training:__call__] E=43 B=171800 s/sec=2312.47 tok/sec=89235.76 u/sec=22.76 ppl=6.889306 [2023-12-17:13:26:06:INFO:sockeye.training:__call__] E=43 B=171850 s/sec=1939.58 tok/sec=98750.42 u/sec=23.36 ppl=6.879438 [2023-12-17:13:26:08:INFO:sockeye.training:__call__] E=43 B=171900 s/sec=2287.26 tok/sec=95040.06 u/sec=23.91 ppl=6.871347 [2023-12-17:13:26:10:INFO:sockeye.training:__call__] E=43 B=171950 s/sec=2241.51 tok/sec=95951.73 u/sec=23.74 ppl=6.863835 [2023-12-17:13:26:12:INFO:sockeye.training:__call__] E=43 B=172000 s/sec=2061.23 tok/sec=100992.97 u/sec=23.25 ppl=6.855880 [2023-12-17:13:26:12:INFO:sockeye.training:_create_checkpoint] Checkpoint [43] Updates=172000 Epoch=43 Samples=15329344 Time-cost=172.005 Updates/sec=23.255 [2023-12-17:13:26:12:INFO:sockeye.training:_create_checkpoint] Checkpoint [43] Train-ppl=6.855880 [2023-12-17:13:26:15:INFO:sockeye.training:_evaluate] Checkpoint [43] Validation-ppl=30.971902 Validation-bleu=0.161286 Validation-chrf=30.904963 Validation-rouge1=0.159615 Validation-rouge2=0.075155 Validation-rougel=0.152279 Validation-length-ratio-mse=1.282078 Validation-ter=96.491952 Validation-avg-sec-per-sent=0.003414 Validation-decode-walltime=1.706895 [2023-12-17:13:26:15:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 19 checkpoints, best so far: 32.780032 [2023-12-17:13:26:15:INFO:sockeye.training:_determine_convergence] Sufficient improvement to continue: 0.796795 > 0.000000 over 20 checkpoints [2023-12-17:13:26:15:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [43] Learning-rate=0.000131 [2023-12-17:13:26:16:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:13:26:16:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=3, base_lr=0.0002, lr=0.00013122000000000003, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:13:26:16:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00043" [2023-12-17:13:26:18:INFO:sockeye.training:__call__] E=43 B=172050 s/sec=727.77 tok/sec=36448.56 u/sec=8.52 ppl=6.228267 [2023-12-17:13:26:20:INFO:sockeye.training:__call__] E=43 B=172100 s/sec=2226.37 tok/sec=97688.75 u/sec=23.58 ppl=6.280219 [2023-12-17:13:26:22:INFO:sockeye.training:__call__] E=43 B=172150 s/sec=1994.81 tok/sec=100758.26 u/sec=22.88 ppl=6.271898 [2023-12-17:13:26:24:INFO:sockeye.training:__call__] E=43 B=172200 s/sec=2187.29 tok/sec=98461.51 u/sec=23.29 ppl=6.298535 [2023-12-17:13:26:27:INFO:sockeye.training:__call__] E=43 B=172250 s/sec=2142.19 tok/sec=97854.60 u/sec=23.82 ppl=6.326034 [2023-12-17:13:26:29:INFO:sockeye.training:__call__] E=43 B=172300 s/sec=2310.43 tok/sec=96735.89 u/sec=23.60 ppl=6.343915 [2023-12-17:13:26:31:INFO:sockeye.training:__call__] E=43 B=172350 s/sec=1981.45 tok/sec=99057.49 u/sec=23.37 ppl=6.331440 [2023-12-17:13:26:33:INFO:sockeye.training:__call__] E=43 B=172400 s/sec=2119.31 tok/sec=98252.22 u/sec=23.40 ppl=6.341256 [2023-12-17:13:26:35:INFO:sockeye.training:__call__] E=43 B=172450 s/sec=2150.87 tok/sec=96990.05 u/sec=23.26 ppl=6.348259 [2023-12-17:13:26:37:INFO:sockeye.training:__call__] E=43 B=172500 s/sec=2103.38 tok/sec=96710.79 u/sec=23.23 ppl=6.359650 [2023-12-17:13:26:39:INFO:sockeye.training:__call__] E=43 B=172550 s/sec=2115.00 tok/sec=101047.51 u/sec=23.07 ppl=6.366293 [2023-12-17:13:26:42:INFO:sockeye.training:__call__] E=43 B=172600 s/sec=1772.48 tok/sec=102136.15 u/sec=22.94 ppl=6.366681 [2023-12-17:13:26:44:INFO:sockeye.training:__call__] E=43 B=172650 s/sec=1994.13 tok/sec=100483.34 u/sec=23.12 ppl=6.379698 [2023-12-17:13:26:46:INFO:sockeye.training:__call__] E=43 B=172700 s/sec=1982.18 tok/sec=99860.81 u/sec=23.03 ppl=6.382433 [2023-12-17:13:26:48:INFO:sockeye.training:__call__] E=43 B=172750 s/sec=2203.82 tok/sec=98088.36 u/sec=23.03 ppl=6.396126 [2023-12-17:13:26:50:INFO:sockeye.training:__call__] E=43 B=172800 s/sec=2174.87 tok/sec=96319.04 u/sec=23.81 ppl=6.400281 [2023-12-17:13:26:52:INFO:sockeye.training:__call__] E=43 B=172850 s/sec=2044.71 tok/sec=100461.74 u/sec=23.19 ppl=6.406406 [2023-12-17:13:26:55:INFO:sockeye.training:__call__] E=43 B=172900 s/sec=1985.59 tok/sec=101709.87 u/sec=23.15 ppl=6.412292 [2023-12-17:13:26:57:INFO:sockeye.training:__call__] E=43 B=172950 s/sec=2042.05 tok/sec=102233.24 u/sec=22.71 ppl=6.418584 [2023-12-17:13:26:59:INFO:sockeye.training:__call__] E=43 B=173000 s/sec=2147.15 tok/sec=93954.87 u/sec=23.88 ppl=6.426932 [2023-12-17:13:27:01:INFO:sockeye.training:__call__] E=43 B=173050 s/sec=2296.81 tok/sec=94970.27 u/sec=23.73 ppl=6.439954 [2023-12-17:13:27:03:INFO:sockeye.training:__call__] E=43 B=173100 s/sec=2119.95 tok/sec=95345.16 u/sec=23.41 ppl=6.450039 [2023-12-17:13:27:05:INFO:sockeye.training:__call__] E=43 B=173150 s/sec=1813.65 tok/sec=99910.21 u/sec=22.67 ppl=6.454536 [2023-12-17:13:27:07:INFO:sockeye.training:__call__] E=43 B=173200 s/sec=1946.76 tok/sec=98764.84 u/sec=23.22 ppl=6.456337 [2023-12-17:13:27:10:INFO:sockeye.training:__call__] E=43 B=173250 s/sec=1966.67 tok/sec=99437.41 u/sec=23.15 ppl=6.463498 [2023-12-17:13:27:12:INFO:sockeye.training:__call__] E=43 B=173300 s/sec=2049.54 tok/sec=100584.08 u/sec=23.29 ppl=6.467665 [2023-12-17:13:27:14:INFO:sockeye.training:__call__] E=43 B=173350 s/sec=2178.20 tok/sec=98302.93 u/sec=23.04 ppl=6.477452 [2023-12-17:13:27:16:INFO:sockeye.training:__call__] E=43 B=173400 s/sec=2023.03 tok/sec=99832.72 u/sec=23.41 ppl=6.479887 [2023-12-17:13:27:18:INFO:sockeye.training:__call__] E=43 B=173450 s/sec=1981.94 tok/sec=103569.86 u/sec=22.73 ppl=6.483331 [2023-12-17:13:27:20:INFO:sockeye.training:__call__] E=43 B=173500 s/sec=1983.12 tok/sec=100034.94 u/sec=23.08 ppl=6.490029 [2023-12-17:13:27:23:INFO:sockeye.training:__call__] E=43 B=173550 s/sec=2093.78 tok/sec=99019.06 u/sec=23.16 ppl=6.495565 [2023-12-17:13:27:25:INFO:sockeye.training:__call__] E=43 B=173600 s/sec=2000.76 tok/sec=97600.90 u/sec=23.51 ppl=6.501228 [2023-12-17:13:27:27:INFO:sockeye.training:__call__] E=43 B=173650 s/sec=2033.99 tok/sec=98862.51 u/sec=23.45 ppl=6.507297 [2023-12-17:13:27:29:INFO:sockeye.training:__call__] E=43 B=173700 s/sec=1907.02 tok/sec=104825.60 u/sec=22.28 ppl=6.509706 [2023-12-17:13:27:31:INFO:sockeye.training:__call__] E=43 B=173750 s/sec=2067.90 tok/sec=100356.31 u/sec=23.16 ppl=6.516763 [2023-12-17:13:27:33:INFO:sockeye.training:__call__] E=43 B=173800 s/sec=2002.66 tok/sec=98305.94 u/sec=22.39 ppl=6.521311 [2023-12-17:13:27:36:INFO:sockeye.training:__call__] E=43 B=173850 s/sec=2410.92 tok/sec=92202.37 u/sec=24.15 ppl=6.533350 [2023-12-17:13:27:38:INFO:sockeye.training:__call__] E=43 B=173900 s/sec=1896.18 tok/sec=100590.91 u/sec=23.37 ppl=6.534556 [2023-12-17:13:27:40:INFO:sockeye.training:__call__] E=43 B=173950 s/sec=2022.81 tok/sec=101608.29 u/sec=23.20 ppl=6.542008 [2023-12-17:13:27:42:INFO:sockeye.training:__call__] E=43 B=174000 s/sec=1965.51 tok/sec=100307.87 u/sec=23.18 ppl=6.546760 [2023-12-17:13:27:44:INFO:sockeye.training:__call__] E=43 B=174050 s/sec=2087.48 tok/sec=102153.52 u/sec=23.13 ppl=6.551892 [2023-12-17:13:27:46:INFO:sockeye.training:__call__] E=43 B=174100 s/sec=1818.13 tok/sec=101815.46 u/sec=22.91 ppl=6.556052 [2023-12-17:13:27:49:INFO:sockeye.training:__call__] E=43 B=174150 s/sec=2212.80 tok/sec=103411.78 u/sec=22.34 ppl=6.561441 [2023-12-17:13:27:51:INFO:sockeye.training:__call__] E=43 B=174200 s/sec=1910.36 tok/sec=100961.39 u/sec=23.05 ppl=6.567744 [2023-12-17:13:27:53:INFO:sockeye.training:__call__] E=43 B=174250 s/sec=2167.54 tok/sec=96854.84 u/sec=23.89 ppl=6.572402 [2023-12-17:13:27:55:INFO:sockeye.training:__call__] E=43 B=174300 s/sec=1900.60 tok/sec=98357.02 u/sec=22.46 ppl=6.576038 [2023-12-17:13:27:57:INFO:sockeye.training:__call__] E=43 B=174350 s/sec=1982.37 tok/sec=102405.27 u/sec=23.03 ppl=6.580490 [2023-12-17:13:27:59:INFO:sockeye.training:__call__] E=43 B=174400 s/sec=2158.25 tok/sec=96726.24 u/sec=23.75 ppl=6.590258 [2023-12-17:13:28:02:INFO:sockeye.training:__call__] E=43 B=174450 s/sec=2017.21 tok/sec=102604.23 u/sec=22.80 ppl=6.597032 [2023-12-17:13:28:04:INFO:sockeye.training:__call__] E=43 B=174500 s/sec=2035.85 tok/sec=100705.71 u/sec=22.64 ppl=6.601711 [2023-12-17:13:28:06:INFO:sockeye.training:__call__] E=43 B=174550 s/sec=2294.65 tok/sec=97581.37 u/sec=23.71 ppl=6.607517 [2023-12-17:13:28:08:INFO:sockeye.training:__call__] E=43 B=174600 s/sec=2015.30 tok/sec=102089.48 u/sec=22.37 ppl=6.611036 [2023-12-17:13:28:10:INFO:sockeye.training:__call__] E=43 B=174650 s/sec=2236.82 tok/sec=98074.02 u/sec=23.50 ppl=6.617879 [2023-12-17:13:28:12:INFO:sockeye.training:__call__] E=43 B=174700 s/sec=1929.72 tok/sec=100270.03 u/sec=23.51 ppl=6.620801 [2023-12-17:13:28:14:INFO:sockeye.training:__call__] E=43 B=174750 s/sec=2273.13 tok/sec=96090.94 u/sec=23.33 ppl=6.624834 [2023-12-17:13:28:17:INFO:sockeye.training:__call__] E=43 B=174800 s/sec=1871.75 tok/sec=103167.08 u/sec=23.03 ppl=6.628269 [2023-12-17:13:28:19:INFO:sockeye.training:__call__] E=43 B=174850 s/sec=2003.11 tok/sec=99668.52 u/sec=22.72 ppl=6.632921 [2023-12-17:13:28:21:INFO:sockeye.training:__call__] E=43 B=174900 s/sec=2202.50 tok/sec=95795.52 u/sec=23.86 ppl=6.637675 [2023-12-17:13:28:23:INFO:sockeye.training:__call__] E=43 B=174950 s/sec=1999.84 tok/sec=100508.67 u/sec=23.06 ppl=6.642272 [2023-12-17:13:28:25:INFO:sockeye.training:__call__] E=43 B=175000 s/sec=2020.56 tok/sec=102108.42 u/sec=23.13 ppl=6.647497 [2023-12-17:13:28:27:INFO:sockeye.training:__call__] E=43 B=175050 s/sec=1749.04 tok/sec=101797.16 u/sec=22.97 ppl=6.650322 [2023-12-17:13:28:30:INFO:sockeye.training:__call__] E=43 B=175100 s/sec=2085.96 tok/sec=98382.30 u/sec=23.49 ppl=6.655827 [2023-12-17:13:28:32:INFO:sockeye.training:__call__] E=43 B=175150 s/sec=1921.60 tok/sec=102051.12 u/sec=22.62 ppl=6.660390 [2023-12-17:13:28:34:INFO:sockeye.training:__call__] E=43 B=175200 s/sec=2158.54 tok/sec=96402.17 u/sec=23.96 ppl=6.667240 [2023-12-17:13:28:36:INFO:sockeye.training:__call__] E=43 B=175250 s/sec=2283.39 tok/sec=97838.07 u/sec=23.63 ppl=6.673964 [2023-12-17:13:28:38:INFO:sockeye.training:__call__] E=43 B=175300 s/sec=2110.26 tok/sec=103976.17 u/sec=22.55 ppl=6.678846 [2023-12-17:13:28:40:INFO:sockeye.training:__call__] E=43 B=175350 s/sec=2120.02 tok/sec=97626.65 u/sec=23.58 ppl=6.684197 [2023-12-17:13:28:42:INFO:sockeye.training:__call__] E=43 B=175400 s/sec=2245.57 tok/sec=97887.45 u/sec=23.51 ppl=6.690673 [2023-12-17:13:28:45:INFO:sockeye.training:__call__] E=43 B=175450 s/sec=2181.77 tok/sec=98281.59 u/sec=23.63 ppl=6.696090 [2023-12-17:13:28:47:INFO:sockeye.training:__call__] E=43 B=175500 s/sec=2164.61 tok/sec=101644.90 u/sec=22.89 ppl=6.700966 [2023-12-17:13:28:49:INFO:sockeye.training:__call__] E=43 B=175550 s/sec=2324.69 tok/sec=95132.44 u/sec=23.94 ppl=6.707584 [2023-12-17:13:28:51:INFO:sockeye.training:__call__] E=43 B=175600 s/sec=1823.61 tok/sec=103388.26 u/sec=22.48 ppl=6.710000 [2023-12-17:13:28:53:INFO:sockeye.training:__call__] E=43 B=175650 s/sec=1943.47 tok/sec=99814.13 u/sec=23.18 ppl=6.714193 [2023-12-17:13:28:55:INFO:sockeye.training:__call__] E=43 B=175700 s/sec=2068.99 tok/sec=99221.32 u/sec=23.47 ppl=6.718784 [2023-12-17:13:28:58:INFO:sockeye.training:__call__] E=43 B=175750 s/sec=2185.17 tok/sec=98234.57 u/sec=23.63 ppl=6.725010 [2023-12-17:13:29:00:INFO:sockeye.training:__call__] E=44 B=175800 s/sec=2068.07 tok/sec=97370.31 u/sec=21.80 ppl=6.723902 [2023-12-17:13:29:02:INFO:sockeye.training:__call__] E=44 B=175850 s/sec=2020.30 tok/sec=96854.37 u/sec=23.47 ppl=6.716118 [2023-12-17:13:29:04:INFO:sockeye.training:__call__] E=44 B=175900 s/sec=2022.73 tok/sec=100884.79 u/sec=23.15 ppl=6.707155 [2023-12-17:13:29:06:INFO:sockeye.training:__call__] E=44 B=175950 s/sec=2125.03 tok/sec=95802.81 u/sec=23.97 ppl=6.699314 [2023-12-17:13:29:08:INFO:sockeye.training:__call__] E=44 B=176000 s/sec=2096.48 tok/sec=97939.13 u/sec=23.11 ppl=6.691794 [2023-12-17:13:29:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [44] Updates=176000 Epoch=44 Samples=15685328 Time-cost=172.473 Updates/sec=23.192 [2023-12-17:13:29:08:INFO:sockeye.training:_create_checkpoint] Checkpoint [44] Train-ppl=6.691794 [2023-12-17:13:29:11:INFO:sockeye.training:_evaluate] Checkpoint [44] Validation-ppl=31.778880 Validation-bleu=0.172071 Validation-chrf=31.356439 Validation-rouge1=0.163102 Validation-rouge2=0.080321 Validation-rougel=0.156992 Validation-length-ratio-mse=1.278343 Validation-ter=95.418902 Validation-avg-sec-per-sent=0.003258 Validation-decode-walltime=1.629090 [2023-12-17:13:29:11:INFO:sockeye.training:_determine_improvement] Validation-chrf has not improved for 20 checkpoints, best so far: 32.780032 [2023-12-17:13:29:11:INFO:sockeye.training:_determine_convergence] Maximum number of not improved checkpoints reached: improvement 0.000000 <= 0.000000 over 20 checkpoints [2023-12-17:13:29:11:INFO:sockeye.training:_adjust_learning_rate] Checkpoint [44] Learning-rate=0.000131 [2023-12-17:13:29:12:INFO:sockeye.training:_save_optimizer_state] Saved optimizer state to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/optimizer_last.pkl" [2023-12-17:13:29:12:INFO:sockeye.training:_save_lr_scheduler] Saved 'LearningRateSchedulerPlateauReduce(reduce_factor=0.90, reduce_num_not_improved=8, num_not_improved=4, base_lr=0.0002, lr=0.00013122000000000003, warmup=0, warmed_up=True)' to '/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/tmp.training_state/lr_scheduler_last.pkl' [2023-12-17:13:29:12:INFO:root:save_parameters] Saved params/state_dict to "/shares/volk.cl.uzh/amoryo/checkpoints/sockeye/expanded-cleaned/model/params.00044" [2023-12-17:13:29:12:INFO:sockeye.training:fit] Training finished. Best checkpoint: 24. Best validation chrf: 32.780032